AINewsnow

run MoE models bigger than your RAM, from phones to GPUs, with one command. Benchmark

I'm working on running mixture-of-experts models that don't fit in memory. Only a few experts are active per token, so it streams them from the SSD and keeps the most used ones in RAM and on the GPU. It's built on llama.cpp's public API: any MoE GGUF llama.cpp loads works, no conversion. --auto mea…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

  1. 2026-10-30 15:01 · r/LocalLLM
    run MoE models bigger than your RAM, from phones to GPUs, with one command. Benchmark

More stories

  1. LFM3-1.7b seems impressive — r/LocalLLaMA
  2. 769 tok/s on a Single RTX 4090: Spark-X2.6-1.7B DSpark — DEV Community — Machine Learning
  3. Halving the KV Cache Doubles Capacity and Buys 1% of Speed — DEV Community — Machine Learning
  4. Are bespoke inference engines overhyped? My Strata experience on 2× R9700 — r/LocalLLaMA
  5. How we fit a 27B model on a 12 GB laptop GPU — DEV Community — Machine Learning
  6. Run a 35B to 180B LLM Without a GPU: VIDRAFT POCKET Passes 3.15 Million Downloads — DEV Community — Machine Learning
  7. How to Run LFM3-Omni-2B GGUF Locally (Only llama.cpp Makes It Talk Back) — DEV Community — Machine Learning
  8. A game scripting language that runs speech-to-text 5x faster than llama.cpp on CPU — Lobsters AI

Get the daily brief of stories like this at 6:30 every morning →