Developer fine-tunes DiffusionGemma into an abstract text-to-image generator
A developer successfully fine-tuned DiffusionGemma—a text diffusion model built on Gemma—into a crude image generation and manipulation system using a small amount of data and compute. By repurposing its output heads and prefilling prompts, the model was coaxed into outputting native visual latent representations. This lighthearted experiment showcases the flexibility of experimental architectures like text diffusion models, demonstrating that they can be creatively repurposed outside their original design parameters. It offers a glimpse into how hobbyists leverage LLM-assisted pipelines to explore unintended capabilities in open-weights models. The project involved training a vision decoder to interpret the model's soft tokens, patching an upstream bug in Transformers for image attention, and using block autoregressive generation to improve image manipulation tasks.
## BACKGROUND
DiffusionGemma is an experimental open-weights generative AI model released by Google DeepMind that utilizes parallel discrete diffusion for text generation instead of traditional token-by-token autoregression. Text diffusion models adapt iterative noise-refinement processes originally popularized in image generation to create and process text data much faster.