llama.cpp b10791 Released with OpenCL Kernel Optimizations
llama.cpp release b10791 introduces targeted OpenCL performance optimizations for quantized LM head execution, decode GEMV, and medium-batch GEMM operations. These updates specifically enhance execution speeds for speculative decoding and Multi-Token Prediction (MTP) workloads across supported hardware. This update improves edge and mobile LLM inference efficiency on OpenCL-compatible processors such as Qualcomm Adreno chips. Optimizing memory-bandwidth-bound decode GEMV and batch GEMM kernels directly leads to faster token generation on low-power devices. Key changes include split-K execution support for q8_0 and q4_K decode GEMV kernels and gating of fused q4_K matrix multiplication with GLU specifically for Adreno GPUs. Additionally, the release restricts certain tiled GEMV defaults to specific architectures based on empirical performance measurements and resolves layout issues during tensor read-backs.
## BACKGROUND
During LLM inference, the decode phase generates tokens sequentially and heavily relies on General Matrix-Vector (GEMV) operations, which are often memory-bandwidth bound. Techniques like Multi-Token Prediction (MTP) and speculative decoding generate multiple candidate tokens in parallel, shifting workloads toward medium-batch General Matrix Multiply (GEMM) operations. OpenCL provides an open framework that enables llama.cpp to execute these heavy mathematical operations efficiently on diverse GPU hardware beyond NVIDIA CUDA.