llama.cpp Release b10907 Fixes KV Cache Allocation for Multi-Token Prediction Architectures
llama.cpp has released patch version b10907, which resolves Key-Value (KV) cache allocation issues in Multi-Token Prediction (MTP) contexts. The fix specifically targets models utilizing DeepSeek-V2, GLM-4-MoE, and Cohere-2-MoE architectures. Correct KV cache allocation is essential to prevent memory corruption and crashes when executing multi-token prediction on complex Mixture-of-Experts (MoE) models. This patch ensures stable local inference performance and speculative decoding support across these major open-weights model families. The release introduces inverse architecture gating and comprehensive architecture testing to handle MTP layer filtering properly. This change ensures MTP-enabled layers allocate memory accurately during context setup across supported MoE architectures.
## BACKGROUND
llama.cpp is an open-source C/C++ inference framework designed to run large language models efficiently on local hardware. Multi-Token Prediction (MTP) is an inference optimization technique where a model predicts multiple future tokens simultaneously rather than generating one token at a time. The KV cache stores key and value representations of previous tokens to prevent redundant memory computations during generation.