llama.cpp Releases Build b10312 to Prevent Eviction of Busy Models
The llama.cpp project has released build b10312, which introduces a fix to prevent the server router from evicting busy models. This update addresses a specific routing issue in the server component to improve stability. This release ensures stability for multi-model serving environments by preventing active models from being prematurely unloaded during inference. It helps maintain consistent performance and reduces latency spikes caused by reloading busy models. The release includes pre-built binaries for various platforms including macOS, Linux, Windows, and Android, supporting CPU, CUDA, Vulkan, and OpenVINO. However, macOS Apple Silicon builds with KleidiAI enabled and openEuler builds are currently disabled in this release.
## BACKGROUND
llama.cpp is a popular open-source software library written in C/C++ that enables efficient local and cloud-based inference on large language models (LLMs). It serves as the core engine for many widely used local AI tools, such as Ollama and LM Studio. The library is designed to run LLMs with minimal setup and high performance across a wide range of hardware architectures.