llama.cpp Adds Support for NVIDIA's Nemotron-3-Puzzle-75B Hybrid Model
Pull Request #25444 to llama.cpp adds support for NVIDIA's Nemotron-3-Puzzle-75B-A9B, a hybrid architecture combining Mamba, Mixture-of-Experts (MoE), and Attention layers. The update enables local execution of this model, which compresses the architecture down to 75.3 billion total parameters and 9.3 billion active parameters. This pull request expands llama.cpp's capabilities to handle complex, non-uniform hybrid neural network architectures on consumer hardware. It brings NVIDIA's compressed 75B model to open-source inference tooling without requiring enterprise-grade hardware. To accommodate Puzzle-75B's non-uniform design across its 40 MoE layers, llama.cpp was updated to support per-layer expert feed-forward lengths and routing top-k counts via array-based GGUF metadata loading. The update also adds support for parsing block configurations for Multi-Token Prediction (MTP) sub-blocks.
## BACKGROUND
Mamba is a State Space Model (SSM) architecture that provides linear scaling for long sequences, avoiding the memory bottlenecks of traditional Attention mechanisms. Mixture-of-Experts (MoE) dynamically routes input tokens to a subset of specialized expert networks to keep computational overhead low. Combining Mamba, MoE, and Attention allows hybrid models like Nemotron-3 to maintain high task accuracy while significantly lowering inference costs.