llama.cpp Release b10758 Brings Hexagon DSP Matrix Fusion and Memory Optimizations
llama.cpp release b10758 introduces major Hexagon DSP performance updates, including QKV and FFN matrix multiplication fusion for Qualcomm's HMX hardware. It also adds virtual address defragmentation and buffer sorting to prevent memory allocation failures during inference. These optimizations significantly improve local LLM execution efficiency and stability on Qualcomm Snapdragon devices powered by Hexagon DSPs. By lowering memory overhead and reducing hardware instruction calls, edge AI applications can achieve higher inference throughput with lower power consumption. The update removes a hardcoded sub-32K dimension restriction, replaces scalar divisions with fastdiv, and tightens overhead sizing to ensure large dimensions do not exceed the Vector Tightly-Coupled Memory (VTCM) budget. Additionally, MUL_MAT_ID operations are fused into MUL_MAT_ID_NX to streamline operation queues on HMX.
## BACKGROUND
Qualcomm Hexagon is a family of digital signal processors (DSPs) and NPUs found in Snapdragon mobile chips, featuring Hexagon Matrix Extensions (HMX) specifically designed for hardware-accelerated tensor computations. Efficient LLM inference on edge hardware often requires utilizing Vector Tightly-Coupled Memory (VTCM), an ultra-fast on-chip memory space, which requires strict memory budgeting to avoid crashes.