inclusionAI Releases Ling-3.0-flash-VL Multimodal MoE Model with 1M Context
inclusionAI has released Ling-3.0-flash-VL, an open-weights multimodal Mixture-of-Experts (MoE) model with 124B total parameters and 5.5B active parameters per token. The model extends language reasoning capabilities to native image and video understanding across a 1-million-token context window. By pairing high parameter capacity with sparse activation and long-context vision-language reasoning, the model enables efficient processing of extended videos and complex agentic workflows. This architecture demonstrates how hybrid attention mechanisms can significantly cut memory costs while maintaining top-tier multimodal performance. The architecture incorporates a ViT visual encoder with a 2-layer MLP projector, VideoRoPE for spatial-temporal encoding, and a 42-layer backbone alternating KDA and Gated MLA layers at a 5:1 ratio. This novel design supports detailed event localization, long-video question answering, and clip editing with high inference efficiency.
## BACKGROUND
Mixture-of-Experts (MoE) models use specialized sub-networks to process inputs, routing tokens to specific experts to keep computation low despite a large overall parameter count. KDA (Kimi Delta Attention) and Gated MLA (Multi-Latent Attention) are linear and low-rank attention mechanisms designed to dramatically reduce memory consumption and KV-cache bottleneck in long sequences. VideoRoPE adapts rotary position embeddings to 3D video data by explicitly capturing both temporal sequence and spatial coordinates.