AINewsnow

Budget rig for local LLM inference + distillation: 2x RTX 3060 12GB vs 2x Tesla P100 16GB?

Hey everyone, I'm building a budget rig (India, so used-market pricing) to run a ~27B model locally with llama.cpp, and I also want to try distillation (teacher -> small student, 2x RTX 3060 12GB (~Rs. 25k each): 24GB total, bf16, Flash Attention, QLoRA/Unsloth support 2x Tesla P100 16GB (~Rs. 20k…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

  1. 2026-10-19 13:37 · r/LocalLLM
    Budget rig for local LLM inference + distillation: 2x RTX 3060 12GB vs 2x Tesla P100 16GB?

More stories

  1. I don't get the qwen 3.8 flash next hype — r/LocalLLaMA
  2. V100 16GB: A 2017 GPU running a 35B vision model at 93 tok/s: Tesla V100 16 GB on Windows — r/LocalLLM
  3. DeepSeek V4.1 Flash 1017 now available on AI Gateway — Vercel Blog
  4. Qwen 27B (IQ4_XS, Q8/Q4 KV) at full 262k context on a 16GB RX 9070 XT: 80+ t/s decode with Adaptive KV Streaming — r/LocalLLM
  5. WHIRL 0.2.0 now runs on the AMD Ryzen AI Max+ 395 (Radeon 8060S): native Windows LLM engine, faster than llama.cpp ROCm and Vulkan from 2k to ~250k tokens (also on the Radeon AI PRO R9700) — r/LocalLLM
  6. Is there an alternative to base llama.cpp for RDNA2 gpus in general with local LLM usage ? — r/LocalLLaMA
  7. llama server's default port changed from 8080 to 9931 — r/LocalLLaMA
  8. Calling all GPUs: help us benchmark SonicBoom v0.0.3 — a compiler-first AI runtime that beats llama.cpp on decode — r/LocalLLaMA

Get the daily brief of stories like this at 6:30 every morning →