Meta Releases MobileMoE: Efficient On-Device Mixture-of-Experts Models
Meta has released MobileMoE, a family of on-device Mixture-of-Experts (MoE) language models ranging from 0.3B to 0.9B active parameters (1.3B to 5.3B total parameters). These models are designed to fit within mobile DRAM limits with an INT4 footprint of less than 3 GB. This release advances the feasibility of running high-quality, large-scale language models locally on mobile devices without relying on cloud infrastructure. By utilizing the MoE architecture, these models optimize the trade-off between computational efficiency and model performance. MobileMoE is available in three scales (S, M, and L) and three variants per scale: Base, SFT (supervised fine-tuning), and QAT (quantization-aware training). The largest model, MobileMoE-L, features 922M active parameters, 60 routed experts (with 4 active per token), and Grouped-Query Attention (GQA).
## BACKGROUND
Mixture-of-Experts (MoE) is a machine learning technique that routes input tokens to specialized sub-networks ("experts") rather than activating the entire model, reducing compute requirements. Grouped-Query Attention (GQA) optimizes memory usage by grouping query heads to share key-value heads. Quantization-Aware Training (QAT) models the effects of low-precision quantization during training to maintain accuracy when the model is compressed.