ggerganov/llama.cpp Released Build b11790 with Hexagon Optimizations
Release b11790 of llama.cpp introduces NEON-vectorized and threaded repack optimizations specifically designed for Qualcomm Hexagon architectures. It adds vectorized q4_0 repacking, extends support to other data types, and implements simple chunking and threading. These incremental improvements enhance local large language model inference performance on mobile and edge devices powered by Snapdragon chips. By optimizing how data is processed and formatted for hardware accelerators like the Hexagon NPU, users can experience faster and more efficient on-device AI execution. The update incorporates changes from pull request #30901, co-authored by contributors from Qualcomm. It features targeted micro-optimizations that vectorize remaining types like q1, q2, and f16 alongside the primary q4_0 format.
## BACKGROUND
llama.cpp is a widely used open-source library that enables efficient inference of large language models on local hardware. Qualcomm Hexagon is a brand of digital signal processor and neural processing unit (NPU) integrated into Snapdragon platforms to accelerate workloads with high power efficiency.