Cursor Open-Sources Mixture-of-Kittens (MoK) for Faster MoE Training on NVL72
Cursor has open-sourced Mixture-of-Kittens (MoK), a deterministic Mixture-of-Experts (MoE) training megakernel designed for NVIDIA NVL72 systems. The kernel fuses all MoE communication and computation, achieving speeds up to 2.37x faster than the strongest public baselines. Training large-scale MoE models is highly resource-intensive, and communication overhead between GPUs often acts as a major bottleneck. By significantly accelerating training on NVIDIA's latest NVL72 architecture, MoK can reduce training times and costs for advanced AI models. MoK operates as a single, fully deterministic megakernel, which helps eliminate kernel boundaries and ensures reproducible training runs. It is specifically optimized for NVL72 rack-scale systems, which feature liquid cooling and connect Grace CPUs with Blackwell GPUs.
## BACKGROUND
Mixture-of-Experts (MoE) is a machine learning architecture that uses routing to direct inputs to specific "expert" sub-networks, improving efficiency. A megakernel is a GPU optimization technique that fuses multiple operations and communication steps into a single kernel launch to minimize overhead. The NVIDIA GB200 NVL72 is a rack-scale computing system connecting 36 Grace CPUs and 72 Blackwell GPUs using NVLink.