DeepSeek Reportedly Training 2-Trillion Parameter Model with Plans for 8-Trillion Model
Unverified social media reports indicate that DeepSeek is currently training a 2-trillion parameter large language model, with long-term ambitions to scale up to an 8-trillion parameter model. Following DeepSeek's recent disruptions in AI training efficiency, pushing into multi-trillion parameter scales signals their ambition to compete directly with premier proprietary AI labs. If realized, scaling such models could significantly advance the capabilities of open-weights AI architectures. The reports build on DeepSeek's existing Mixture-of-Experts (MoE) design, where total parameter count is high but active parameters per token remain much lower to maintain inference efficiency. However, these claims stem from unconfirmed online rumors and currently lack official documentation or benchmark results from DeepSeek.
## BACKGROUND
Large language models often use a Mixture-of-Experts (MoE) architecture, which replaces standard feed-forward networks with multiple specialized sub-networks or 'experts'. During inference, a router activates only a small subset of these experts for each token, allowing models to hold trillions of total parameters while running with the speed and memory overhead of a much smaller model.