CachyLLama: A llama.cpp fork featuring persistent SSD-backed KV caching
CachyLLama is a newly released fork of llama.cpp designed to accelerate local AI agent workflows by implementing persistent, SSD-backed Key-Value (KV) caching and multi-tier state management. It allows prompt evaluation states to survive system restarts and power cycles, drastically reducing warm-start times for large prompts. Local agentic workflows often suffer from high latency due to repetitive prompt evaluation of system prompts and conversation history. By caching these states to disk, CachyLLama eliminates redundant processing overhead, making local LLM execution viable and fast on mid-tier hardware and APUs. The tool supports hybrid MoE/SSM architectures like DeepSeek-V3 and Qwen 3.5, and benchmark tests show a prompt processing speedup from 143.1 seconds (cold start) to 0.99 seconds (cached) for a 15,700-token prompt. However, it only accelerates prompt processing and does not speed up the actual token generation phase.
## BACKGROUND
Key-Value (KV) caching is a technique used in LLM inference to store previously calculated attention states, preventing the need to recompute them for every new token. In agentic workflows, where system instructions and chat history are repeatedly sent to the model, recalculating these tokens (prompt evaluation) is often the primary bottleneck.