inclusionAI Releases Ling-3.0-tiny, a Fast Local MoE Model
The Ling team has released Ling-3.0-tiny, an open-weight Mixture of Experts (MoE) model featuring 8 billion total parameters and 1.3 billion active parameters. This model is designed for high-speed local inference, following their recent release of the larger Ling-3.0-flash. Tiny MoE models offer a compelling balance for local LLM deployment by providing the performance of larger models while maintaining a low memory footprint and high generation speeds. This release makes efficient, high-throughput local AI assistant capabilities more accessible on consumer hardware like MacBooks. In FP8 precision, Ling-3.0-tiny achieves speeds of 100-105 tokens/second on DGX Spark and 86-90 tokens/second on an M4 Pro MacBook. It requires approximately 8.34 GiB of peak memory at an 8K context length, with performance positioned between the 4B and 8-12B Qwen and Gemma models.
## BACKGROUND
A Mixture of Experts (MoE) architecture uses a routing mechanism to direct inputs to only a subset of its total parameters (active parameters) for each token. This allows the model to run at the speed of a smaller model (based on active parameters) while retaining the capacity and knowledge of a larger model (based on total parameters), though it still requires memory to load the entire model.