Training a 1.1B Parameter LLM From Scratch for $200
A developer successfully trained a 1.1B parameter Gemma-based LLM from scratch on 20 billion tokens for approximately $200, subsequently fine-tuning it with LoRA. The creator has open-sourced the code, model weights in safetensors and GGUF formats, and hosted a live demo. This project demonstrates that pretraining custom, small-scale language models from scratch is highly accessible and cost-effective for individual developers. It provides a practical blueprint and open-source resources for budget-friendly AI experimentation and learning. The model's architecture is based on Gemma 3 but modified with a shorter context length of 4096, no sliding window attention, and a smaller 32k vocabulary. Pretraining was done on vast.ai using an H100 GPU for 130 hours, followed by LoRA fine-tuning on an RTX 3060 for 52 hours.
## BACKGROUND
Pretraining a Large Language Model (LLM) from scratch involves training it on a massive corpus of text to learn general language patterns, which typically requires significant compute. LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that allows developers to adapt these pretrained models to specific tasks, like chat, at a fraction of the computational cost.