Running the 2.8T Kimi-k3 Model Locally via llama.cpp
A user successfully ran the massive 2.8-trillion-parameter Kimi-k3 model locally on a high-end workstation using a pending llama.cpp pull request (PR #26185) to convert and run the model in GGUF format. However, the generation speed was extremely slow, averaging just 0.23 tokens per second. This achievement demonstrates the feasibility of running frontier-class Mixture of Experts (MoE) models locally, providing valuable hardware benchmarks and configuration baselines for the open-source AI community. It highlights the growing capability of consumer and workstation hardware to handle models that previously required massive cloud infrastructure. The setup utilized a Threadripper PRO 9965WX CPU, 512 GB of DDR5 RAM, and two RTX 6000 GPUs, with the model split across CPU and GPU memory. To improve performance in the future, the user plans to distribute the workload across a cluster using llama.cpp's RPC server over a high-speed network.
## BACKGROUND
Kimi-k3 is a frontier-class large language model featuring 2.8 trillion parameters and a Mixture of Experts (MoE) architecture, which dynamically routes tasks to specialized sub-networks. llama.cpp is an open-source project designed for efficient LLM inference on local hardware, and its RPC server feature allows multiple machines to pool their hardware resources for distributed inference.