Baidu Open-Sources ERNIE 5.5 Flash Base 280B Model
Baidu has open-sourced ERNIE 5.5 Flash Base, a 280B parameter sparse Mixture of Experts (MoE) language model trained from scratch on 33.5T tokens. The model supports a context length of 128K during training and up to 1M tokens during fine-tuning and inference. This release provides researchers and developers with valuable pre-trained weights for domain adaptation, instruction tuning, and reinforcement learning research. It serves as a foundational milestone before the subsequent release of the fully post-trained ERNIE 5.5 Flash model. The model activates 14B parameters per token and utilizes a distributed Muon optimizer along with FP8 mixed precision. It was evaluated across 30 benchmark tasks and is made available in both FP8 and BF16 formats on Hugging Face.
## BACKGROUND
A sparse Mixture of Experts (MoE) is a neural network architecture where only a subset of parameters, known as experts, is activated for any given input, allowing for high model capacity with efficient computation. FP8 mixed precision utilizes 8-bit floating-point formats to accelerate training and reduce memory consumption while maintaining model accuracy.