AMD Releases Instella-MoE, a 16B Open-Source Model Trained on Own Hardware
AMD has announced the Instella-MoE series, a collection of 16-billion-parameter open-source Mixture of Experts (MoE) models. The models were trained from scratch entirely on AMD Instinct MI300X and MI325X GPUs using the ROCm software stack. This release demonstrates AMD's capability to train competitive, state-of-the-art MoE models end-to-end using its own hardware and software ecosystem. It promotes hardware diversity in the AI industry, offering a viable alternative to Nvidia's dominant CUDA platform. The model features 2.8 billion active parameters, a 27-layer decoder architecture, and incorporates Gated Multi-Head Latent Attention (Gated MLA). The training utilized the FarSkip-Collective library to speed up pretraining by 12.7%, while the SGLang inference framework reduced Time to First Token (TTFT) by up to 39.2%.
## BACKGROUND
Mixture of Experts (MoE) is a machine learning technique that routes inputs to specialized 'expert' subnetworks, allowing models to scale parameters while keeping computational costs low. AMD's ROCm is an open-source software platform for GPU-accelerated computing, serving as AMD's primary competitor to Nvidia's proprietary CUDA platform.