llama.cpp b10693 Adds Qualcomm Hexagon NPU Device Discovery and Lazy Allocation
The llama.cpp project has released version b10693, introducing runtime device discovery and lazy session allocation for Qualcomm Hexagon NPUs. This update also includes early rejection of non-existent devices during the initialization phase. This release improves hardware utilization and efficiency for local AI execution on Snapdragon-powered edge devices. By dynamically detecting NPU cores and allocating sessions only when needed, it reduces memory overhead and startup times. The update implements lazy session allocation and cleanup interfaces for Hexagon devices, alongside runtime discovery of available NPU cores. It rejects non-existent devices early in the initialization process to prevent runtime crashes.
## BACKGROUND
llama.cpp is a popular open-source project that enables efficient LLM inference on consumer hardware using C/C++. Qualcomm Hexagon NPUs are specialized hardware accelerators integrated into Snapdragon processors, designed to speed up machine learning tasks on mobile devices and laptops. Lazy allocation is a programming pattern that delays the creation of a resource until it is actually needed, saving system memory.