llama.cpp b10739 Released with Metal Flash-Attention Vector Tuning for M2 Max
Release b10739 of llama.cpp introduces Metal Flash-Attention vector tuning specifically optimized for Apple M2 Max GPUs with 30 cores. The update incorporates benchmarked tuning parameters for f16 and q8_0 data types directly into the Metal backend lookup table. This optimization improves LLM inference speed and resource efficiency on Apple Silicon Macs, particularly for devices powered by the M2 Max chip. It highlights the open-source community's ongoing efforts to squeeze maximum performance out of specific hardware configurations. The tuning values were generated using the 'ggml-metal-tuning fa-vec' utility for f16 and q8_0 precision and committed into 'fa_vec_tuned_table' via PR #28015. Pre-built release binaries were updated across multiple OS targets, including macOS, Linux, Windows, and Android.
## BACKGROUND
llama.cpp is a high-performance C/C++ framework for running LLMs locally across diverse hardware, using Apple's Metal API for GPU acceleration on macOS. FlashAttention is an IO-aware algorithm designed to speed up transformer attention layers by minimizing slow memory reads and writes between high-bandwidth memory and fast on-chip SRAM.