NVIDIA Releases Dory1.0-4B-BF16: A 4B Looped LLM with 256K Context
NVIDIA has released Dory1.0-4B-BF16, a 4B parameter large language model utilizing a looped architecture with a normalized-GPT design. The model features a 256K token context window and combines sliding-window attention with full-attention layers. This release represents an important architectural shift toward compute-efficient small language models that scale intelligence by repeatedly applying shared layer blocks. By leveraging a looped design, small models can achieve higher reasoning capabilities without a proportional increase in physical parameter size. Dory1.0-4B-BF16 supports English general-purpose reasoning and is released under the OpenMDW License Agreement, version 1.1, allowing for both commercial and non-commercial use. It uses interleaved hybrid sliding-window attention to efficiently manage long contexts up to 256K tokens.
## BACKGROUND
A looped architecture, or latent-recurrent model, repeatedly passes information through a shared block of layers to iteratively refine representations, allowing models to spend more compute per token at test time. Sliding-window attention is a sparse self-attention mechanism that limits each token to attending only to a fixed local window, reducing computational complexity from quadratic to linear for long-context processing.