New Tool "cache-pressure" Validates Local LLM KV Cache Eviction under Pressure
Developer "co-l" released `cache-pressure`, an open-source tool and validation protocol designed to stress-test local LLM serving engines like vLLM, llama.cpp, and SGLang under context pressure. The tool calibrates cache hit and miss expectations, hydrates contexts to fill memory, and measures actual KV cache retention and eviction behavior. Advertised KV cache capacities reported by inference engines often differ from real-world retained tokens when memory is fully saturated. This tool provides developers and engine maintainers with empirical ground truth to benchmark deployments, validate cache bug fixes, and optimize serving configurations. During initial tests on vLLM, applying specific deduplication and bound-fix patches boosted context retention from 51.98% (1.05M tokens) to 146.56% (3.00M tokens) on an advertised 2M capacity. The validation process works by filling the cache with stable contexts and probing in reverse order to identify exactly when and which old contexts are evicted.
## BACKGROUND
In Large Language Model (LLM) inference, Key-Value (KV) caching stores intermediate attention states of processed tokens to prevent redundant calculations during text generation. Serving frameworks such as vLLM use high-throughput memory management algorithms to handle these caches, but when VRAM limits are reached, cache eviction policies determine which contexts are cleared from memory.