~/DIFFUSION MO/nvidia-releases-nemotron-labs-sigma-8b-and-3b-continuous-diffusion-language-models

NVIDIA Releases Nemotron-Labs-Sigma 8B and 3B Continuous Diffusion Language Models

NVIDIA has introduced Nemotron-Labs-Sigma-8B-instruct and 3B models, which are continuous diffusion language models designed for parallel text generation. Instead of traditional autoregressive token generation, they denoise Gaussian noise in a 16-dimensional continuous token-embedding space using DDPM or DDIM samplers. This release represents a significant architectural shift in the open-source LLM space away from traditional token-by-token autoregression toward parallel block decoding. It enables faster text generation speeds through block-parallel sampling while maintaining competitive reasoning accuracy. The models use a Ministral-style decoder-only Transformer architecture combined with a continuous-diffusion generation head supporting an 8,192-token context length. They are implemented in plain PyTorch and load via standard transformers without requiring custom kernels or training frameworks.

## BACKGROUND

Continuous diffusion language models apply concepts from image-generative diffusion models to text by operating in a continuous embedding space rather than discrete token categories. Denoising Diffusion Probabilistic Models (DDPM) and Denoising Diffusion Implicit Models (DDIM) are iterative sampling techniques used to progressively clean noise to generate coherent outputs.

## REFERENCES

## KEYWORDS

#Diffusion Models#Large Language Models#NVIDIA#Machine Learning#Open Source AI

$ subscribe --daily