ggerganov/llama.cpp Released Version b11953 with CUDA MoE Fusion
llama.cpp release b11953 introduces CUDA top-k Mixture of Experts (MoE) fusion optimized for 384 experts, along with standard cross-platform binary distributions. The release also includes precompiled packages across various backends like Vulkan, OpenVINO, and ROCm for major operating systems. This performance optimization significantly accelerates the execution of large Mixture of Experts language models on NVIDIA GPUs. It enables local AI practitioners to run complex, multi-expert models more efficiently and with lower latency. The main technical addition is pull request #31288, which adds CUDA support for top-k MoE fusion tailored for architectures featuring 384 experts. The repository continues to offer extensive pre-built binaries for macOS, Linux, Windows, and Android environments.
## BACKGROUND
llama.cpp is a popular open-source library that allows running large language models locally on consumer hardware through the GGML tensor library. Mixture of Experts (MoE) is a machine learning technique where only a subset of sub-networks, or 'experts', are activated for each token to boost efficiency.