llama.cpp Release b12314 Adds Decomposed Flash Attention for Adreno GPUs
llama.cpp release b12314 introduces decomposed Flash Attention (FA) optimizations and kernel refinements for OpenCL on Adreno GPUs. It expands support for smaller head dimensions and batch sizes while optimizing memory read and write operations. This release significantly improves the performance and efficiency of running large language models on mobile and edge devices equipped with Qualcomm Adreno GPUs. It enhances local AI capabilities by reducing memory overhead during the attention mechanism execution. The update lowers the default head dimension threshold from 256 to 128 on X2E, adds support for dp4a instructions for PV matrix multiplication, and enables direct PV by default to avoid V transposition for small batches.
## BACKGROUND
llama.cpp is an open-source library that enables efficient inference of large language models like Llama using C and C++. Flash Attention is an algorithm designed to speed up the attention mechanism in transformers and reduce memory usage by utilizing tiling to minimize reads and writes to slow device memory.