NInfer fork: 555k context@fp4 for 5090 with YARN, reliable kv host cacheing, monitoring, jinja, opened model support
This story is from 2026-09-05. It is preserved in the archive; the latest stories are on the live feed.
Hiya, NInfer is amazng for Qwen, but lacking for real-world-use. As adoption of issues/pr's was not really what I needed, I created a fork and hit it for this week with 3 concurrent claude code session until it didn't break any longer. Hope you like it. NVFP4 KV cache (from scratch) I implemented a…
Read the full story at r/LocalLLaMA ↗
Timeline · 1 report
- 2026-09-05 22:58 · r/LocalLLaMA
NInfer fork: 555k context@fp4 for 5090 with YARN, reliable kv host cacheing, monitoring, jinja, opened model support