llama.cpp Release b10662 Introduces Unified KV Cache Configuration Argument
The llama.cpp project has released version b10662, which introduces the `--kv-unified-per-slot` (or `ctx-per-slot`) server argument. This update refactors the unified KV cache configuration by replacing context fractions with integer-based context pool slots. Efficient KV cache management is critical for reducing memory bottlenecks during LLM inference, especially in multi-user serving scenarios. This change simplifies how developers allocate context memory per slot, potentially improving the stability and performance of llama.cpp servers. The pull request (#24124) was co-authored by Xuan Son Nguyen from Hugging Face and refactors the configuration to make `ctx-per-slot` an integer. Additionally, this release disables KleidiAI acceleration for macOS Apple Silicon in the default builds.
## BACKGROUND
During LLM inference, the Key-Value (KV) cache stores attention keys and values of past tokens to avoid redundant calculations, but it acts as a major memory bottleneck. Llama.cpp is a popular open-source inference engine designed to run LLMs efficiently on consumer hardware and servers.