I just built a mini Kimi-K3 from Scratch under 250$. Already beats GPT-2 (124M)!
This story is from 2026-08-20. It is preserved in the archive; the latest stories are on the live feed.
I pre-trained a 1.02-billion-parameter on Kimi K3 replica trained on 5.00 billion decontaminated tokens for $250. This model has 1.02 billion parameters, of which 145 million are active per token. It is roughly one two-thousandth of K3 by total size. It saw 5,000,003,584 tokens, which is a rounding…
Read the full story at r/LocalLLaMA ↗
Timeline · 1 report
- 2026-08-20 11:38 · r/LocalLLaMA
I just built a mini Kimi-K3 from Scratch under 250$. Already beats GPT-2 (124M)!