Maxwell-V1-Flash: A 35B Mixture-of-Experts Model with 256K Context Support
The Maxwell Team has released Maxwell-V1-Flash, a 35B parameter Mixture-of-Experts (MoE) language model featuring 3B active parameters per token and support for context lengths up to 256K tokens. It introduces architectural innovations including Maxwell Block Sparse Attention and Manifold-Constrained Hyper-Connections. This release provides the local AI community with a high-performance, cost-efficient open-weights model capable of advanced reasoning, agentic tasks, and extreme long-context processing. By integrating cutting-edge sparse attention and multi-stream routing techniques, it sets a new benchmark for practical efficiency in modern LLM architecture. The model uses Maxwell Block Sparse Attention to bound main attention to 4,096 tokens per KV group—achieving a 6.7x speedup over dense GQA at 256K context—and relies on Manifold-Constrained Hyper-Connections to widen information flow without adding extra attention overhead.
## BACKGROUND
Mixture-of-Experts (MoE) is a transformer architecture design where only a sparse subset of neural network parameters (experts) is activated for each incoming token, allowing models to scale up total capacity while keeping inference costs low. Manifold-Constrained Hyper-Connections (mHC) is a recently proposed machine learning framework that extends residual connections by projecting hyper-connection spaces onto a specific manifold to restore identity mapping properties.