Optimizing Moonshot AI's 2.8T Kimi K3 Model on a 16x GB10 Cluster
Developer /u/ciprianveg successfully benchmarked and ran Moonshot AI's 2.8-trillion parameter Kimi K3 model across a 16-node GB10 cluster. The customized setup achieved sustained coding decode speeds of 30 tokens/second and prefill throughput ranging from 750 to 910 tokens/second. Deploying a 2.8-trillion parameter model traditionally presents severe networking and runtime bottlenecks. Demonstrating high throughput and multi-user concurrency on private hardware shows how custom vLLM stacks and kernel tweaks make ultra-large open models practical for real-world tasks. The setup utilizes a dual MikroTik switch layout with 4x 400G-to-4x100G breakout cables optimized for NCCL network topology. On the software side, it relies on a patched `gb10-vllm` stack with custom Multi-Head Latent Attention (MLA) and KV cache kernels to handle extended context workflows without memory starvation.
## BACKGROUND
Kimi K3 is an open-weight flagship Mixture-of-Experts (MoE) large language model developed by Moonshot AI featuring 2.8 trillion parameters. NVIDIA Collective Communications Library (NCCL) topology tuning is critical when scaling multi-GPU and multi-node clusters to maintain fast inter-node data transfer during LLM inference.