AINewsnow

Running 100B+ MoE Models on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp

This story is from 2026-10-05. It is preserved in the archive; the latest stories are on the live feed.

Tuần này trên Hacker News có một bài hơn 600 điểm: chạy một model MoE 125B tham số trên một con RTX 4090 mà vẫn đạt tốc độ sinh token rất đáng nể. Nghe như chuyện đùa, vì 4090 chỉ có 24GB VRAM, trong khi 125B tham số ở mức quantize 4-bit đã chiếm khoảng 70-75GB. Bí quyết không nằm ở phép màu nào cả…

Read the full story at DEV Community — Machine Learning ↗

Timeline · 1 report

  1. 2026-10-05 01:58 · DEV Community — Machine Learning
    Running 100B+ MoE Models on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp

More stories

  1. How to Speed Up Qwen3.8-27B in llama.cpp (NVIDIA's Draft Model May Lose to MTP) — DEV Community — Machine Learning
  2. Thinking On or Off? Gemma 4 E2B on an 8 GB RTX 3050 — DEV Community — AI
  3. Running 26B Coding Models Locally on a 6GB Laptop GPU at 45 Tokens/Sec with MAPLE and llama.cpp — DEV Community — Machine Learning
  4. Cutting LLM Fine-Tuning Memory in Pure Kotlin: Fused Linear Cross-Entropy and the JVM Vector API — DEV Community — Machine Learning
  5. RAMDeck Architecture Analysis: llama.cpp RPC, Distributed LLM Memory, and SBC Compatibility — DEV Community — Machine Learning
  6. llama.cpp finally has a 1.0 release. Check your old integrations before upgrading. — r/artificial
  7. Rotate K/V read order for better L2 cache hit rates by gaugarg-nv · Pull Request #32407 · ggml-org/llama.cpp — r/LocalLLaMA
  8. Signal #136: Qwen4-Max, llama.cpp v1.0.0, and the Git Patch You Need Today — DEV Community — AI

Read the Daily Brief: the day's most important AI stories →