llama.cpp Gets Major CUDA Improvements: FP8 Support and Faster GPU Inference

The llama.cpp project introduces OCP FP8 support, FP8 KV-cache capabilities and targeted CUDA optimisations. Here's what the latest developments mean for local AI and GPU inference.

Read the full story at r/LocalLLM ↗

Timeline · 1 report

2026-11-12 10:58 · r/LocalLLM
llama.cpp Gets Major CUDA Improvements: FP8 Support and Faster GPU Inference

More stories

Llama App - Windows’ta ücretsiz indir ve yükle — Microsoft Research Blog
KV Cache Math: Llama 3.1 8B Needs 16 GiB Just to Hold 128K Tokens — DEV Community — Machine Learning
Gemma4-31b latent space discovery: Caring and Contempt for user controlled by the same 3 neurons! Plus, how to find your own steering recipes and export to GGUF steering vector for llama.cpp — r/LocalLLaMA
Ten engines for one box: which local inference engine fits a 100 GB rig with two 16 GB cards? — DEV Community — Machine Learning
Open-sourcing TokenShark: Step through local LLMs block-by-block, live-edit the flow in your browser, and trace where model thinks — r/LocalLLM
Local LLMs for Agentic Coding: A Practical Guide for Modest Hardware — DEV Community — AI
Qwen3.6-35B-A3B Q4_K_M (llama.cpp) or Qwen3.8-Flash-Next Coder IQ1_M (Strata) for coding agent? — r/LocalLLM
Kinetic: a new open-source LLM engine in Rust. Runs your GGUFs on Windows and Linux, replaces llama-server with the same flags — r/LocalLLM

Get the daily brief of stories like this at 6:30 every morning →