llama.cpp Release b10790 Tunes CUDA Kernel Crossover for NVIDIA Orin (SM87)
The llama.cpp project released version b10790, introducing PR #28285 to tune the crossover threshold between MMVQ and MMQ CUDA kernels on SM87 hardware. This automated release also updates pre-built binaries across Linux, Windows, macOS, Android, and iOS platforms. This optimization improves CUDA inference efficiency for users running quantized Large Language Models on NVIDIA Jetson Orin edge devices. Refining the transition point between vector and matrix kernels allows the runtime to maintain higher throughput during prompt ingestion and token generation. The change specifically targets NVIDIA SM87 architecture (Compute Capability 8.7) by optimizing when ggml-cuda switches from Matrix-Matrix-Vector Quantized (MMVQ) to Matrix-Matrix Quantized (MMQ) execution. Updated release artifacts support multiple compute backends, including CUDA 12/13, Vulkan, ROCm, OpenVINO, and SYCL.
## BACKGROUND
llama.cpp is a widely used C/C++ inference framework designed to run LLMs efficiently on diverse hardware. In its CUDA backend, MMVQ handles matrix-vector operations during single-token generation, whereas MMQ handles matrix-matrix operations for larger batch sizes and prompt processing. NVIDIA SM87 refers to the GPU compute capability powering the Jetson AGX Orin and Orin Nano system-on-modules.