Agnes-3.0-Flash: A 33B Multimodal Hybrid LLM with 262K Context Window
Agnes-AI has introduced Agnes-3.0-Flash, an open 33-billion parameter multimodal model capable of understanding text, images, and video over a 262,144-token context window. The model adopts a 3:1 hybrid architecture that alternates between 54 gated delta-rule recurrent layers and 18 standard global attention layers. Standard global attention models suffer from massive Key-Value (KV) cache memory overhead when handling long context windows. By offloading sequence tracking to gated delta-rule layers in 75% of its architecture, Agnes-3.0-Flash significantly reduces memory consumption while scaling context capacity efficiently. Only 18 of the model's 72 decoder layers require a dynamic KV cache, as the 54 gated delta-rule layers utilize fixed-size recurrent states independent of sequence length. It also integrates a 27-layer vision tower projected to a 5120-dimension hidden space and uses 3-axis mROPE (text, height, width) position embeddings for multimodal input.
## BACKGROUND
Traditional Transformer architectures rely on global self-attention, where memory footprint grows linearly with prompt length, creating hardware bottlenecks for long-context generation. Hybrid architectures resolve this issue by pairing global attention with linear attention or recurrent mechanisms (such as gated delta rules), maintaining steady memory usage across most layers.