llama.cpp Release b10868 Adds IQ Quantization Support for MoE Models
llama.cpp release b10868 introduces IQ (Importance Matrix) quantization type handling for Mixture of Experts (MoE) models via PR #28476. This update enables efficient low-bit quantization for complex MoE architectures, allowing users to run large sparse models locally with reduced RAM and VRAM usage. The automated build includes compiled binaries for multiple operating systems and backends, including macOS, Windows, Linux, Android, CUDA 12/13, Vulkan, ROCm, and SYCL.
## BACKGROUND
llama.cpp is a widely used C/C++ library designed for local LLM inference across diverse hardware. Quantization compresses high-precision model weights (like FP16) into lower bit-widths, where IQ (Importance Matrix) quantization uses calibration data to retain output quality at ultra-low bitrates. Mixture of Experts (MoE) is a neural network design that routes input tokens to specialized sub-networks, reducing computational costs compared to dense models.