llama.cpp b10357 Released with OpenCL FlashAttention Optimization
llama.cpp has released version b10357, which introduces an OpenCL optimization that transposes the Key (K) tile in local memory for FlashAttention prefill kernels. This minor release focuses on improving memory access patterns during the initial prompt-processing phase. This optimization enhances the performance of LLM inference on hardware platforms that rely on OpenCL, such as integrated GPUs and non-NVIDIA accelerators. By improving memory efficiency during the prefill phase, users can expect faster processing of input prompts. The update specifically targets the FlashAttention prefill kernels by transposing the K tile in local memory, which helps align memory access and reduce latency. The release also provides pre-built binaries for a wide range of operating systems and hardware backends, including Windows, Linux, macOS, and Android.
## BACKGROUND
LLM inference consists of two main stages: the prefill phase, which processes the input prompt in parallel to build the KV cache, and the decode phase, which generates tokens sequentially. FlashAttention is an optimized algorithm designed to speed up the attention mechanism by reducing memory reads and writes. OpenCL is a framework that allows software to run across heterogeneous platforms, enabling llama.cpp to run efficiently on diverse hardware beyond standard NVIDIA GPUs.