llama.cpp Release b11821 Optimizes Intel XMX Matrix Engines via SYCL
llama.cpp release b11821 refactors multi-column support for Intel XMX matrix engines on the SYCL backend. It introduces fused kernels and optimizations for speculative decoding verification and short prompt batches. This update significantly improves inference performance and hardware acceleration on Intel GPUs for advanced workloads like speculative decoding. It benefits users running large language models on Intel hardware by increasing token generation speeds. The release adds a fused gate/up kernel to compute feed-forward network weights more efficiently and includes two new environment variables for controlling maximum columns. On the B70 hardware, performance for Q4_K_M improved by 5.9% to reach 28.23 tokens per second.
## BACKGROUND
llama.cpp is a popular open-source software library designed to run large language models efficiently across various hardware platforms. SYCL is a cross-platform programming model that allows developers to run code across heterogeneous accelerators such as Intel GPUs, while Intel XMX refers to hardware matrix extension units built into Intel graphics architectures to accelerate AI computations.