ExLlamaV3 Update Adds MoE CPU Offloading and Self-Calibrated Quantization
Developer turboderp released major updates to ExLlamaV3, introducing CPU offloading for Mixture-of-Experts (MoE) experts and ngram disk offloading for Qwen-3.8-Flash-Next. The update also adds support for GLM-5.3-Flash alongside a new self-calibrated quantization optimization technique. Offloading MoE expert weights to system RAM allows users to run massive multi-billion parameter models on consumer GPUs with limited VRAM. These updates significantly expand the capacity and efficiency of local LLM inference for open-source AI enthusiasts. The release showcases EXL3 quantization optimizations down to 3.05 bits per weight (bpw) on Qwen-3.8-Flash-Next, demonstrating high precision tasks like SVG code generation. It also implements disk offloading for ngrams to further decrease system memory overhead during generation.
## BACKGROUND
ExLlamaV3 is an open-source inference engine optimized for running large language models on consumer-grade NVIDIA GPUs. Mixture-of-Experts (MoE) models partition network parameters into specialized sub-networks ('experts'), activating only a subset per token but requiring large total memory. Quantization compresses high-precision model weights into fewer bits (e.g., 3-bit or 4-bit) to drastically reduce memory usage.