ggerganov/llama.cpp Released b11604 with KleidiAI SME2 Support
Llama.cpp release b11604 introduces KleidiAI SME2 Q4_K and Q6_K matrix multiplication support for ARM architectures. This update brings optimized performance routines tailored for specific Arm hardware capabilities. This optimization enhances on-device AI inference speeds for ARM-based CPUs supporting SME2 instructions. It benefits users running large language models on mobile and edge hardware by maximizing processor capabilities. The pull request #29219, contributed by Charles Xu from Arm, integrates KleidiAI routines to accelerate Q4_K and Q6_K quantized model workloads. These changes target matrix multiplication (matmul), a critical performance bottleneck in LLM inference.
## BACKGROUND
KleidiAI is an open-source library that provides performance-critical routines and micro-kernels optimized for Arm CPUs. Quantization schemes like Q4_K and Q6_K in llama.cpp allow large language models to run efficiently on consumer hardware by reducing model size with minimal accuracy loss.