llama.cpp b10643 Released with Multi-NPU Hexagon Support
The llama.cpp b10643 release introduces support for multi-NPU Qualcomm Hexagon devices and implements a fully asynchronous backend. It also adds optimizations like a DMA pipeline, VTCM layout adjustments, and fused ALLREDUCE operations. This update significantly improves LLM inference performance and efficiency on Snapdragon-powered edge devices. By enabling multi-NPU utilization and asynchronous processing, it allows for faster, low-power local AI execution on mobile and edge hardware. The release features a fully asynchronous backend using non-host buffers by default, alongside support for Q8_0 quantization with in-place dequantizers. It also introduces sync tokens to coordinate multiple NPU devices during asynchronous tensor splits.
## BACKGROUND
Qualcomm Hexagon is a family of digital signal processors (DSPs) and Neural Processing Units (NPUs) designed for low-power, high-performance AI acceleration on Snapdragon chips. Vector Tightly Coupled Memory (VTCM) is a high-speed, on-chip memory architecture used in Hexagon processors to accelerate vector operations. llama.cpp is a popular open-source project that enables efficient LLM inference across various hardware backends.