HySparse2 Architecture Paper Released as Foundation for Upcoming MiMo-V3 LLM
The research team behind the MiMo model series has published the paper for HySparse2, a hybrid sparse attention architecture designed to serve as the core framework for MiMo-V3. HySparse2 introduces a novel optimization approach for sparse neural networks centered around two-level Key-Value (KV) cache sharing. As large language models scale to handle increasingly long contexts, memory usage and compute overhead for self-attention become critical bottlenecks. HySparse2 provides a substantially more efficient architecture for long-context reasoning models by drastically cutting down KV cache overhead and memory bandwidth requirements. HySparse2 features two levels of KV sharing: outer-level KV Bridging, which uses a YOCO-style self-decoder and cross-decoder structure to connect full-attention layers, and inner-level KV Reuse, where sparse layers reuse the KV cache and token selection indices from the preceding full-attention layer.
## BACKGROUND
Standard large language models use dense self-attention, storing Key-Value (KV) pairs for every token across all layers, which consumes large amounts of GPU memory during text generation. Sparse attention architectures lower these requirements by computing attention over only a targeted subset of tokens, but achieving efficiency while maintaining context accuracy requires sophisticated KV cache reuse strategies.