I spent a year making old Tesla P100/V100 cards fast for local LLMs. PXA 4.0 is out today: up to 3x the decode and 8x the prompt speed of mainline llama.cpp on the same box, same files. Numbers, method and the losses inside.
Quick disclosure: this is my project. I build it with a small team and a Discord full of people who run it on hardware most guides tell you to throw away. Two years ago I bought a Tesla P100 for about 150 dollars because it had 16 GB of HBM2 and nobody wanted it. Then a second one. Then two V100s.…
Read the full story at r/LocalLLM ↗
Timeline · 1 report
- 2026-10-30 14:13 · r/LocalLLM
I spent a year making old Tesla P100/V100 cards fast for local LLMs. PXA 4.0 is out today: up to 3x the decode and 8x the prompt speed of mainline llama.cpp on the same box, same files. Numbers, method and the losses inside.
More stories
- Musk and Zuckerberg echo Trump's push to rebrand AI as 'super intelligence' on earnings calls. Others do not — CNBC Technology
- I dpo post-trained llama-3.1-8b to be 100% more robust against jailbreaks (and measured what it costs) — r/huggingface
- Janas: a C inference engine that runs MoE models bigger than your RAM by streaming experts from NVMe, plus an almost complete OpenAI-compatible server, MCP tools, vision and speech (GPL-3.0) — r/LocalLLM
- Arc Pro B60 on WSL2 (Vulkan via Mesa Dozen) is 8 to 40 times slower than native Windows Vulkan, and Qwen3.8-27B segfaults inside Intel's LLVM. Anyone seen this? — r/LocalLLM
- Local Qwen 3.8 27B as the worker, Opus as the planner: same 38 tests passed, half the API bill — r/LocalLLM
- Why does my local LLM keep reverting to generic prose despite extensive author-style examples and prompt engineering? Am I overengineering this? — r/LocalLLM
- I tested a 12B open-source AI Humanizer locally. Here's what I found. — r/learnmachinelearning
- I trained LLaMa from scratch — r/deeplearning
Get the daily brief of stories like this at 6:30 every morning →