How I Built an LLM Inference Engine From Scratch in CUDA (235 tokens/s on a 6 GB Laptop GPU)
My laptop's RTX 3050 now generates 235 tokens per second with TinyLlama-1.1B, using an inference engine where every GPU kernel is written by hand. That is 77% of the fastest this GPU could possibly go, and it took three rounds of measure, find the worst part, rewrite. The project is tinyinfer : abo…
Read the full story at DEV Community — Machine Learning ↗
Timeline · 1 report
- 2026-11-14 18:41 · DEV Community — Machine Learning
How I Built an LLM Inference Engine From Scratch in CUDA (235 tokens/s on a 6 GB Laptop GPU)