llama.cpp b10201 Released with WebGPU Flash Attention Optimizations
llama.cpp has released build b10201, introducing performance improvements for WebGPU flash attention (flash_attn_vec) when using quantized Key-Value (KV) caches. This update specifically targets and optimizes execution in long-context scenarios. Running large language models directly in web browsers via WebGPU often faces memory and performance bottlenecks. By optimizing quantized KV caches for long contexts, this release enables more efficient and scalable local LLM inference on web platforms. The release integrates pull request #25956, which refines the flash_attn_vec implementation for quantized KV caches and resolves related bugs. Additionally, KleidiAI support for macOS Apple Silicon (arm64) remains disabled in this build.
## BACKGROUND
WebGPU is a modern web API that allows browsers to access GPU acceleration for computation. In LLM inference, the Key-Value (KV) cache stores past token representations to avoid redundant calculations, but it grows with context length. Quantization reduces the memory footprint of this cache, while Flash Attention optimizes memory access patterns to speed up attention computation.