llama.cpp b11168 Enhances Qualcomm Hexagon Dynamic Quantization Backend
Release b11168 of llama.cpp introduces dynamic quantizer improvements for Qualcomm Hexagon hardware, including accuracy fixes for Q8_0 matrix multiplication (`MUL_MAT`). It also transitions all dynamic quantization paths to use Direct Memory Access (DMA) and fixes register spills. These optimizations improve inference accuracy and efficiency when running Large Language Models on edge devices equipped with Qualcomm Snapdragon processors. Leveraging DMA across dynamic quantization paths reduces CPU overhead and improves data transfer speeds during model execution. Technical updates include routing all activation data through DMA, removing the obsolete `run_quant_task`, refactoring matrix multiplication operations (`matmul-ops`), and cleaning up `dma_addr_t` casts across the Hexagon Tensor Processor (HTP) codebase.
## BACKGROUND
llama.cpp is a popular open-source C/C++ framework designed for high-performance LLM inference across diverse hardware architectures. Qualcomm Hexagon is a specialized Digital Signal Processor (DSP) integrated into Snapdragon chips, optimized for executing AI workloads and matrix operations efficiently on mobile and edge devices.