llama.cpp Release b11927 Adds Fused Top-K Mask Kernel for Metal
Release b11927 of llama.cpp introduces the GGML_OP_TOP_K_MASK kernel for the Apple Metal backend, porting performance optimizations previously exclusive to CUDA. This implementation utilizes a grid-stride kernel to initialize the output and a threadgroup per row to copy the required mask values. This update helps bridge the performance gap between Apple Silicon and NVIDIA hardware for local large language model inference. Mac users running models locally can expect improved efficiency during specific attention or filtering operations in the generation pipeline. Function constants are utilized to omit unused scores, blocks, and extra paths, reducing computational overhead for execution graphs. The optimization was tested and assisted using a MacBook with an M5 Max chip and 128GB of memory running a Qwen model variant.
## BACKGROUND
llama.cpp is an open-source software library designed to perform efficient inference on large language models using minimal setup across various hardware backends. The Metal backend allows llama.cpp to harness Apple Silicon GPUs natively, while GGUF serves as the standard file format for distributing quantized models.