llama.cpp Release b12069 Adds OpenCL Support for iq4_xs Quantization
llama.cpp release b12069 introduces initial OpenCL support for iq4_xs quantization, adding optimizations specifically tailored for Qualcomm Adreno GPUs. The update includes code contributions from Qualcomm engineers to improve GEMV, GEMM, and batch decoding performance. This release enhances the efficiency of running large language models on mobile devices and hardware powered by Adreno GPUs. It allows users to leverage low-bit quantization formats for better performance and reduced memory footprints on a wider range of hardware. The pull request implements iq4_xs AOS and SOA matrix-vector products, noshuffle GEMM/GEMV, and dp4a instructions for Adreno GPUs. These changes help accelerate inference workloads directly on mobile platforms running OpenCL.
## BACKGROUND
llama.cpp is a popular framework for running large language models locally with minimal setup, utilizing various hardware backends. Quantization techniques like iq4_xs compress model weights to around 4.25 bits per weight, shrinking file sizes while maintaining a high level of accuracy.