AINewsnow

Caution about fast inference engines

I tried a faster inference engine that optimizes for my AMD card. Yes I did get 90-100 tps in generation which is crazy compared to llama.cpp's 45 tps with ROCM. But there is one problem: the optimized engine seems to confuse a lot and half of the time it did not fully understand my intent. I give…

Read the full story at r/LocalLLM ↗

Timeline · 1 report

  1. 2026-11-11 18:28 · r/LocalLLM
    Caution about fast inference engines

More stories

  1. Kinetic: a new open-source LLM engine in Rust. Runs your GGUFs on Windows and Linux, replaces llama-server with the same flags — r/LocalLLM
  2. Run Gemma 4 31B-it Uncensored on a 12GB GPU: Complete Local Vision & 256K Context Guide with llama.cpp, DeepSeek Harness & Hermes Agent — DEV Community — Machine Learning
  3. llama.cpp: Add FP8 KV-cache — r/LocalLLaMA
  4. Study examines whether large language models internally represent psychological constructs — arXiv cs.AI
  5. How do you fine-tune Llama on your own data in the cloud? — DEV Community — Machine Learning
  6. DeepSeek V4.1 Flash API: Clean and Tag 10,000 Comments, and What It Costs — DEV Community — AI
  7. Your coding questions are exactly where AI models disagree most — DEV Community — AI
  8. PolyStrata – the Strata fork that stopped being one engine, and what that means against llama.cpp, Strata and Avalon — DEV Community — AI

Get the daily brief of stories like this at 6:30 every morning →