First Kimi K3 benchmark on home lab achieves 4 tokens per second
A user successfully ran a Q2_K quantized version of the massive 2.8-trillion-parameter Kimi K3 model on a home lab setup, achieving a decoding speed of approximately 4 tokens per second. The setup utilized a custom llama.cpp fork, 768GB of DDR5 RAM, and two Nvidia RTX 5090 GPUs. This demonstrates the feasibility of running a 3T-class open model on high-end consumer/prosumer hardware, bringing frontier-level AI capabilities closer to local deployment. It provides valuable empirical performance data for the local LLM community experimenting with massive models. The user reported a prompt prefill speed of 50-70 tokens per second and noted that the decoding speed increased over time, suggesting a potential warmup or memory swapping effect. However, standard benchmarking via llama-bench crashed during the test.
## BACKGROUND
Kimi K3 is an open 2.8-trillion-parameter large language model featuring a 1-million-token context window, designed for complex coding and reasoning tasks. To run such a massive model on consumer hardware, users rely on quantization techniques like Q2_K, which compresses model weights to save memory at the cost of some precision.