AINewsnow

SSD MoE finally viable!

Hey guys, I've made it! A llama.cpp based inference engine that lets you OFFLOAD UPTO 60-70% OF THE MOE MODELS' ROUTED EXPERTS INTO SSD and still hide the streaming latency down to negligibe(I'm achieving 1-3ms added to the time per token due to SSD streaming when 60% is offloaded in SSD)... Essent…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

  1. 2026-11-04 20:57 · r/LocalLLM
    SSD MoE finally viable!

More stories

  1. I built a one-click installer for local LLMs on Windows: it picks a model for your hardware and sets up Ollama — DEV Community — AI
  2. Mistral Large 4 Is Open Weight. Read the License and Do the Memory Math Before You Self-Host — DEV Community — AI
  3. Now that some time has passed: which Qwen3.8 27B quant do you stick with? — r/LocalLLM
  4. Configuring llama.cpp for Enterprise Environments: A Comprehensive Guide to Concurrency and Stability — DEV Community — AI
  5. Independent study tests Direct Nash Optimization for aligning language models — DEV Community — Machine Learning
  6. The 40% Performance Tax of Ollama: Why Developers Are Moving Back to Raw llama.cpp and vLLM — DEV Community — AI
  7. I track how much each provider charges for the same open LLM: Llama 3.3 70B ranges from $0.155 to $0.78 per 1M tokens — DEV Community — AI
  8. [llama.cpp fork]: QFN + [specific quant] + [ultra specific hardware config] = [incredible and irreproducible pp and tg] — r/LocalLLaMA

Get the daily brief of stories like this at 6:30 every morning →