Moore Threads Open-Sources Megatron-MUSA-Titan for Trillion-Parameter MoE Training
Moore Threads has open-sourced Megatron-MUSA-Titan, a large-scale model training infrastructure project built for its MUSA platform. The project is designed specifically for trillion-parameter Mixture of Experts (MoE) model training and reinforcement learning across 10k-card GPU clusters. This open-source release is a major milestone for domestic AI hardware ecosystems, providing essential infrastructure for scaling up trillion-parameter models on large GPU clusters. It bridges the gap between hardware capabilities and advanced distributed training frameworks, enabling high performance and long-term stability for complex AI workloads. Megatron-MUSA-Titan supports blockwise FP8 training to enhance computational efficiency and integrates DeepEP with ACE asynchronous communication engines to accelerate token distribution. Additionally, it coordinates with reinforcement learning frameworks like slime and miles alongside SGLang for sample generation.
## BACKGROUND
MoE (Mixture of Experts) is an architectural design that scales model capacity efficiently by routing tokens to specific sub-networks, while blockwise FP8 training uses fine-grained quantization to reduce memory overhead and speed up computation. MUSA is Moore Threads' proprietary unified system architecture designed for its full-function GPUs.