Pre-Training a 1.02B Mini Kimi-K3 Replica from Scratch for Under $250
A developer successfully pre-trained a 1.02-billion-parameter mini replica of the Kimi-K3 architecture from scratch on 5 billion tokens for less than $250. The model replicates Kimi-K3's core architecture, including Kimi Delta Attention and LatentMoE, and outperforms GPT-2 (124M) on the HellaSwag benchmark. This project demonstrates that individual developers can pre-train relatively large, modern LLM architectures on a highly constrained budget. It lowers the barrier to entry for experimenting with advanced architectural components like Mixture of Experts (MoE) and linear attention mechanisms. The model features 145 million active parameters per token and utilizes Kimi Delta Attention, Gated MLA, and LatentMoE with an auxiliary-loss-free balancer. It achieved a 33.4% score on HellaSwag compared to GPT-2's 28%, using Kimi-K3's original 163,840-token tokenizer.
## BACKGROUND
Kimi-K3 is a state-of-the-art language model architecture that incorporates advanced efficiency techniques. Among these are Kimi Delta Attention (KDA), which optimizes linear attention with fine-grained gating to reduce KV cache usage, and LatentMoE, which projects routing and expert computations into a lower-dimensional latent space to save compute and memory. Multi-head Latent Attention (MLA) is another key technique designed to compress the key-value (KV) cache during inference.