llama.cpp Adds Support for Dots3-Note, a 280B Multimodal MoE Model
A new pull request in llama.cpp adds support for dots3-note, a 280-billion-parameter open-weight Mixture-of-Experts (MoE) model developed by Dots Studio. The model is multimodal, capable of processing text, images, video, and audio inputs to generate text outputs. This integration allows users to run a massive, highly capable multimodal open-weight model locally or on consumer-grade hardware via llama.cpp's optimized execution. It expands the availability of large-scale Mixture-of-Experts models to the open-source AI community. Despite having 280 billion total parameters, dots3-note only activates 16 billion parameters per token, making it computationally efficient. It also features an exceptionally large context window of up to 512,000 tokens and is released under the Apache 2.0 license.
## BACKGROUND
llama.cpp is a popular open-source framework designed to run LLMs locally with high performance using quantization and minimal dependencies. Mixture-of-Experts (MoE) is an AI model architecture that routes inputs to specialized sub-networks, or 'experts', rather than activating the entire model, significantly reducing computational costs during inference.