Open-Weight Release of GLM-5.3-Flash with Hybrid Sparse-Linear Attention
Z.ai has released GLM-5.3-Flash (formerly known as ox-alpha), an open-weight 320B parameter natively multimodal model featuring a novel hybrid sparse-linear attention architecture. The model is trained on a 30-trillion-token multimodal corpus and claims to outperform GLM-5.2 at one-tenth of the serving cost. This release introduces cutting-edge architectural innovations like Kimi Delta Attention (KDA) and Manifold-Constrained Hyper-Connections (mHC) to the open-source community, significantly reducing long-context serving costs and KV cache usage. It demonstrates that highly efficient, large-scale multimodal models can be run cost-effectively while approaching proprietary-level performance on coding and agentic tasks. The model uses a 45-layer layout repeating blocks of three KDA linear attention layers followed by one DeepSeek-style sparse attention layer, activating 18B out of 320B parameters via a Mixture-of-Experts (MoE) design. It supports a context length of up to 1,048,576 tokens, ships with a Multi-Token Prediction (MTP) head for speculative decoding, and is optimized for FP8 inference.
## BACKGROUND
Traditional Transformer models suffer from high computational and memory overhead, specifically KV cache growth, when processing long contexts. To address this, Kimi Delta Attention (KDA) offers a hardware-optimized linear attention mechanism with fine-grained gating, while Manifold-Constrained Hyper-Connections (mHC) stabilize training in deep networks by constraining residual connections to a specific manifold.