llama.cpp Release b12099 Adds Initial Qualcomm Hexagon DSP Support
llama.cpp release b12099 introduces initial Qualcomm Hexagon DSP support for the LIGHTNING_INDEXER operation. This addition brings optimized prefill and decode performance pathways utilizing specialized hardware components. This update significantly improves local large language model inference capabilities on mobile and edge devices powered by Qualcomm Snapdragon processors. It allows developers to achieve better execution speed and efficiency by offloading workloads onto the device's specialized hardware accelerators. The update implements pipelined HMX paths, support for Q4_0 and F32 data types, and canonical double-buffering for DMA pipelines. Additionally, it provides the GGML_HEXAGON_LI_SELECT configuration parameter to explicitly force HMX or HVX execution paths.
## BACKGROUND
Qualcomm Hexagon is a family of digital signal processors (DSP) and neural processing units (NPU) designed to handle demanding signal processing and machine learning tasks efficiently. In large language model runtimes like llama.cpp, prefill refers to processing the initial input prompt, while decode refers to generating subsequent response tokens.