llama.cpp b11006 Adds Qualcomm Hexagon DSP Kernels for Q4_K and Q6_K
Release b11006 of llama.cpp introduces dedicated Q4_K and Q6_K quantization kernels optimized for Qualcomm Hexagon DSPs. The change (PR #28994), co-authored by an engineer from Qualcomm, improves unpack accuracy and execution speed for K-quant models on Hexagon hardware. This hardware-specific acceleration enables large language models using Q4_K and Q6_K K-quants to run significantly faster and more efficiently on Qualcomm Snapdragon chipsets. It lowers latency and power consumption for local, on-device AI inference on mobile and edge devices. The update combines commits for implementing the Q6_K kernel, improving unpacking accuracy on Hexagon, and adding Q4_K kernel support. Pre-built binaries released alongside b11006 cover multiple platforms, including Android arm64 and Windows arm64 with OpenCL Adreno support.
## BACKGROUND
llama.cpp is a widely used open-source framework written in C/C++ designed to run Large Language Models efficiently on consumer hardware. Quantization techniques like K-quants (e.g., Q4_K and Q6_K) compress model weights to 4-bit or 6-bit values to reduce memory usage, while Qualcomm Hexagon DSPs are specialized processors embedded in Snapdragon chips for high-efficiency AI computational workloads.