llama.cpp Release b11135 Adds Cache Deduplication for Hugging Face Draft Models
llama.cpp released automated build b11135, featuring an update to its server component that deduplicates Hugging Face draft models using model cache deduplication. The release also includes a C++ code fix to avoid capturing structured bindings inside lambda functions. This change optimizes RAM and VRAM usage by preventing redundant model reloading when serving speculative decoding workloads with Hugging Face draft models. It helps maintain server efficiency when running multi-model LLM inference pipelines. The update fixes issue #27846 via PR #27934 by integrating `dedup-cache-models` into the server component for draft model loading. It also resolves a C++ compilation issue related to structured binding capture in lambda expressions.
## BACKGROUND
llama.cpp is a popular open-source C/C++ framework for lightweight, high-performance LLM inference across various hardware architectures. Speculative decoding speeds up LLM inference by using a small 'draft' model to quickly generate candidate tokens, which are then verified in parallel by a larger target model. Managing draft models efficiently in memory is critical for maximizing inference speed without exhausting system resources.