AINewsnow

FreeToken vs llama.cpp on one RTX 3090: llama.cpp is 2–3× faster when the MoE fits in VRAM. On gpt-oss-120b (63 GB), FreeToken gets the first token out 7× faster at 32 concurrent users.

This story is from 2026-10-01. It is preserved in the archive; the latest stories are on the live feed.

I've run the benchmark on a RTX 3090, 1024 tokens in / 256 out, concurrency 1–32. If the model fits on vRAM (Gemma-4-26B-A4B, byte-identical GGUF on both engines ): llama.cpp has 2.2–3.2× the throughput and 5–6× faster TTFT. FreeToken 0.1.2 can't keep 4-bit experts in VRAM at all, and it OOM'd at 8…

Read the full story at r/LocalLLaMA ↗

Timeline · 2 reports

  1. 2026-10-01 20:04 · r/LocalLLaMA
    Qwen3.8-27B Q4_K_M on one RTX 3090 + OpenCode: throughput, four coding tasks, and a reasoning-budget failure
  2. 2026-10-01 14:11 · r/LocalLLaMA
    FreeToken vs llama.cpp on one RTX 3090: llama.cpp is 2–3× faster when the MoE fits in VRAM. On gpt-oss-120b (63 GB), FreeToken gets the first token out 7× faster at 32 concurrent users.

More stories

  1. AI Daily Digest · November 14, 2026: NVIDIA Loses Its Founder-CEO, ChatGPT Gets a Wallet, llama.cpp Hits 1.0 — DEV Community — AI
  2. Study audits major LLMs for racial and sex bias in pain management — arXiv cs.CL
  3. Off Grid AI Apps Bring Local LLMs and Image Generation to Phones and Macs — DEV Community — AI
  4. Kindi – fast local inference on low-cost GPUs, and the exact-match rule that makes its numbers trustworthy — DEV Community — Machine Learning
  5. Show HN: 53ON AI Server – OpenAI-compatible server for AMD Ryzen AI Max — DEV Community — AI
  6. I have created a tool to observe what happens inside an LLM. Also allows for the comparison of two quantizations of the same model to see at what point they diverge, token by token — r/LocalLLaMA
  7. Llama.cpp performance improvement over time (Gemma4 models) — r/LocalLLaMA
  8. llama.cpp was copying 643 MB to the CPU per request. Disabling state saving nearly doubled throughput — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →