llama.cpp b11490 Adds Qualcomm Hexagon Buffer Optimizations and Offloading Option
Release b11490 of llama.cpp introduces multi-buffer allocation support (`alloc_buffer_n`) and large tensor splitting for the Qualcomm Hexagon DSP backend. It also adds a new `--no-embd-offload` flag and increases the default dynamic memory buffer to 512MB to fix performance regressions with large Mixture-of-Experts (MoE) models. This update improves LLM inference performance and stability on Snapdragon-powered mobile and edge devices utilizing Qualcomm's Hexagon NPU/DSP. As on-device AI execution grows in importance, optimizing memory buffer handling prevents performance bottlenecks during complex model runs. The `GGML_HEXAGON_MBUF` configuration option was updated to accept three values (`dyn`, `static`, `total`), allowing finer control over memory allocation. Additionally, the `--no-embd-offload` command-line option simplifies execution scripts on Snapdragon devices where embedding offloading is unnecessary.
## BACKGROUND
llama.cpp is a popular open-source C/C++ library designed for efficient local inference of large language models across various hardware backends. Qualcomm Hexagon DSPs are specialized coprocessors built into Snapdragon processors to accelerate signal processing and AI workloads on mobile devices. Optimizing memory buffers on such embedded hardware is crucial because limited memory bandwidth can severely restrict inference speed.