LongCat-Flash-Lite-Sparse Weights Released with 1M Context Window
The weights for LongCat-Flash-Lite-Sparse have been officially released, upgrading the model's native context length support from 256k to 1 million tokens. This version replaces the dense Multi-head Latent Attention (MLA) mechanism with LongCat Sparse Attention (LSA). This release demonstrates the practical application of sparse attention mechanisms to scale context windows efficiently without the quadratic computational bottleneck of standard attention. It provides researchers with an open-source model to study long-context LLM architectures. By transitioning from dense MLA to LSA, the model reduces computational scaling closer to linear by selecting only the most relevant tokens. The model is built upon the LongCat-Flash-Lite architecture, expanding its context capacity fourfold.
## BACKGROUND
Standard attention mechanisms scale quadratically with context length, making ultra-long contexts computationally expensive. Multi-head Latent Attention (MLA) compresses the KV cache to save memory, while LongCat Sparse Attention (LSA), developed by Meituan as an evolution of DeepSeek Sparse Attention, further optimizes this by dynamically selecting relevant tokens to achieve near-linear scaling.