ggerganov/llama.cpp released b12064 with Qualcomm Hexagon optimizations
Llama.cpp release b12064 introduces a tiled HVX transpose path to significantly improve permuted copy and tensor reshaping performance on Qualcomm Hexagon hardware. This update replaces previous slow element-by-element fallback methods with a optimized 2D transpose mechanism using vector instructions. This optimization drastically accelerates tensor operations like channel-first and channel-last permutes commonly used in video VAEs and edge AI models running on mobile and embedded devices. By improving throughput from around 45 MB/s, it helps unlock the full hardware potential of Qualcomm's digital signal processors. The new implementation utilizes double-buffered staged and transposed buffers to overlap input DMA with transposition and output DMA processes. It also introduces specialized handling for packed modes and small dimensions to avoid performance bottlenecks.
## BACKGROUND
Qualcomm Hexagon is a family of digital signal processors (DSPs) featuring Hexagon Vector Extensions (HVX), which provide SIMD capabilities for accelerating heavy math workloads. Vector Tightly-Coupled Memory (VTCM) is a DSP-exclusive high-speed memory accessed directly by HVX instructions to speed up data-intensive operations.