Huawei Open-Sources 505B Parameter openPangu-2.0-Pro MoE Model
Huawei has officially open-sourced its 505-billion-parameter Mixture of Experts (MoE) language model, openPangu-2.0-Pro, releasing its weights, inference code, and technical report. The model is optimized for Huawei's Ascend NPU ecosystem and supports a 512k context window. This release provides the open-source AI community with a massive, production-grade MoE model tailored for non-NVIDIA hardware, boosting the viability of Huawei's Ascend ecosystem. It demonstrates Huawei's commitment to open-source AI while offering developers a powerful alternative for large-scale model deployment. The model features architectural upgrades including Multi-head Latent Attention (MLA) combined with a hybrid DSA+SWA structure, a 4-branch mHC residual connection, and a 3-head Multi-Token Prediction (MTP) module for faster inference. Despite its 505B total parameters, it only activates 18B parameters per token, significantly reducing computational overhead.
## BACKGROUND
Mixture of Experts (MoE) is a machine learning technique that uses multiple specialized subnetworks ('experts') to process different inputs, allowing models to scale parameters without a proportional increase in compute cost. Multi-head Latent Attention (MLA), popularized by models like DeepSeek, compresses the Key-Value (KV) cache to improve inference efficiency. Ascend NPUs are Huawei's proprietary AI processors designed to provide an alternative computing infrastructure to NVIDIA GPUs.