llama.cpp Release b11645 Adds BF16 Metal Flash Attention Kernel
llama.cpp release b11645 introduces a BF16 (bfloat16) variant of the tensor API flash attention kernel specifically designed for Metal on Apple Silicon. Additionally, it reorganizes the tensor flash attention kernels by relocating them into a dedicated file path. This update improves inference performance and memory efficiency for large language models running on Apple Silicon hardware via Apple's Metal framework. It benefits users who rely on local LLM deployment tools built on top of llama.cpp. The update adds the new BF16 tensor API flash attention kernel and refactors the codebase by moving tensor FA kernels to 'kernels/fa_tensor.metal'. It is part of the ongoing incremental updates to optimize low-level execution paths across various hardware backends.
## BACKGROUND
llama.cpp is a popular open-source library that enables efficient inference of large language models in C and C++, serving as the backbone for many local AI tools. Flash attention is an algorithm designed to speed up the attention mechanism in transformers and reduce memory usage by avoiding unnecessary memory reads and writes.