CachyLLama: A llama.cpp Fork with Persistent SSD-Backed KV Cache for Local Agents
CachyLLama is a new fork of llama.cpp that introduces a persistent, SSD-backed multi-tier KV cache and system-prompt checkpoints. This allows local LLM setups to save and restore context states, eliminating the need to reprocess long prompts even after a server restart. In local LLM agent workflows, repeating large system prompts and conversation histories on every request causes significant latency. By caching these states on SSDs, CachyLLama dramatically reduces response times for subsequent queries, making local agent sessions much more practical on consumer hardware. Benchmarks on a 7840U processor show prompt evaluation times dropping from 143.1 seconds (cold) to just 0.99 seconds (warm) for a ~15,700-token prompt. Additionally, the project supports hybrid architectures like Qwen and Gemma, which require complex recurrent state restoration.
## BACKGROUND
Large Language Models (LLMs) generate text token-by-token, requiring the model to process the entire preceding context at each step. A Key-Value (KV) cache stores intermediate computations of previous tokens to avoid redundant calculations, but standard implementations typically lose this cache once the session or server ends.