AINewsnow

489.8 tok/s, QWen2.5 7B on RTX3050 for code edit, reproducible decoding

This is the v0.0.5 update. It's about one thing: speculative decoding that's fast and doesn't change the output. Same RTX 3050 8 GB, Qwen2.5-7B-Instruct Q4_K_M, same GGUF for both runtimes. llama.cpp 43fe9c64281 with -ngl 99 -fa on, using the fastest of llama-server's eight speculative modes (ngram…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-09 13:49 · DEV Community — AI
    489.8 tok/s, QWen2.5 7B on RTX3050 for code edit, reproducible decoding

More stories

  1. How to pay for AI API access with crypto — no card, no subscription, no account — DEV Community — AI
  2. Ollama vs. llama.cpp on the Same GGUF: The 15% Speed Gap Was the Defaults, Not the Engine — DEV Community — AI
  3. Test if your local inference is safe from the web — r/LocalLLM
  4. Probably old news, but any website you have open can reach your local LLM server while you browse — r/LocalLLM
  5. The Open-Source Paradox: Giving Away AI Makes It Safer (and More Dangerous) — DEV Community — AI
  6. Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU — Hacker News Front Page
  7. Zero-Downtime LLM Inference on Kubernetes Spot Instances (~70% GPU Cost Savings) — DEV Community — Machine Learning
  8. The inference engine flood.. but where's the quality? — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →