GPT-2 KV cache from scratch: correctness, latency, and memory measurements [P]
I built GPT-2 small and a KV cache in PyTorch as a learning project, then compared the mathematical savings with measured latency and memory. I have published the implementation and a detailed write-up, and would appreciate feedback on the methodology. Setup: pretrained GPT-2 small (124M), NVIDIA A…
Read the full story at r/MachineLearning ↗
Timeline · 1 report
- 2026-10-18 08:23 · r/MachineLearning
GPT-2 KV cache from scratch: correctness, latency, and memory measurements [P]