AINewsnow

llama.cpp optimizations for specific hardware and models

I've made a few posts recently, just talking about the hardware I've got and the model I use. Pair of P100s, Unsloth Qwen3.6 35b A3B UD Q4 K XL . There's been a ton of talk of model and hardware specific inference engines showing crazy performance numbers on models and hardware that have no busines…

Read the full story at r/LocalLLaMA ↗

Timeline · 1 report

  1. 2026-10-12 20:24 · r/LocalLLaMA
    llama.cpp optimizations for specific hardware and models

More stories

  1. Qwen3.8-Flash-Next (125B) at ~100 tok/s on an M5 Ultra Mac Studio with llama.cpp — r/LocalLLM
  2. Ternary Bonsai 2 27B on a $250 Intel Arc B580: writes a working snake game at 108 t/s, 89 t/s at 115K context (open source) — r/LocalLLM
  3. I clustered two old Android phones to run a 27B LLM locally. Bypassed RAM limits via C++ RPC split and Hadamard Ternary math (No overheating) — r/LocalLLM
  4. Benchmarked Strata 0.1.25 and 0.1.39 against llama.cpp on three cards: an RTX PRO 6000 (96 GB), a 3090 Ti and a 5090 laptop — r/LocalLLM
  5. Qwen 3.8 27B Q4_K_M running on a OnePlus 13 with 24 GB RAM — r/LocalLLM
  6. BrokenML: can agents fix broken ML jobs? — r/LocalLLaMA
  7. strix-llama new branch: prompt processing +21–54% over Vulkan and up to 2.7× over stock ROCm kernel at 128k, same outputs quality — r/LocalLLaMA
  8. Do you actually test open-model inference compatibility? — r/LocalLLaMA

Get the daily brief of stories like this at 6:30 every morning →