llama.cpp Release b10981 Optimizes OpenVINO Stateful Decode and GPU MoE Inference
llama.cpp released tag b10981, bringing key OpenVINO backend optimizations for stateful KV-cache decoding and GPU Mixture-of-Experts (MoE) inference. The update resolves architectural bugs with multi-head attention layouts and sliding-window layers on complex models like Gemma-4 while introducing features such as disk weight buffer spilling and 4-bit requantization targets. These improvements deliver substantial inference performance gains for Intel hardware users running local LLMs via OpenVINO. Stateful decoding over long contexts saw notable generation speedups, with Gemma-4-12B improving from 6.27 to 9.11 tokens per second at an 8,192 context depth. The release applies `pass::KVStateSeqAxis` across multi-head states to remove repetitive reader-side tensor transpositions during decoding, avoiding performance degradation at high context depths. It also adds environment configurations including `GGML_OPENVINO_REQUANT_KQUANT` for 4-bit target selection and `GGML_OPENVINO_SPILL_DIR` to handle memory constraints by spilling weight buffers to disk.
## BACKGROUND
Large language model inference comprises a prefill phase and an autoregressive decode phase, where each newly generated token depends on prior key-value (KV) states stored in memory. OpenVINO is Intel's open-source toolkit designed to optimize and execute AI models across Intel CPUs, GPUs, and NPUs. Techniques like sliding-window attention restrict KV cache size by retaining only recent context tokens, requiring specialized handling when integrated into stateful execution graphs.