MTP Merged into ik_llama.cpp, Doubling Qwen Inference Speed for Coding
Pull Request #2369 has officially merged Multi-Token Prediction (MTP) support for Qwen models into ik_llama.cpp, enabling the model's 2.6B MTP head to self-draft future tokens. This integration doubles code generation inference speed on high-end setups like the RTX 5090 to 90 tokens per second, while also boosting performance on mid-range cards like the 12GB RTX 4070. This update brings native multi-token speculative decoding to mainstream open-source LLM inference tooling without requiring custom model forks or external patches. Because code structure is highly predictable, it achieves 93–99% draft acceptance rates, offering local developers substantial performance gains for coding assistants. The acceleration uses speculative decoding where draft tokens generated by the 2.6B head are verified by the main model, guaranteeing identical output to non-MTP execution. Current caveats include single-slot processing constraints (-np 1) and lower draft acceptance rates (60–65%) on prose or chain-of-thought reasoning outputs.
## BACKGROUND
ik_llama.cpp is an optimized fork of llama.cpp focusing on advanced quantization algorithms and CPU/GPU hybrid inference performance. Multi-Token Prediction (MTP) is an architectural technique where an LLM is trained to predict multiple future tokens simultaneously rather than a single next token, accelerating inference when paired with speculative decoding.