mini-AGI: A Dynamically Growing Continual Learning Model Trained on 8GB VRAM
Developer /u/Another__one released "mini-AGI", an open-source project demonstrating a dynamically expanding language model currently at 530 million parameters trained from scratch on an 8GB VRAM laptop. The system combines dynamic Mixture-of-Experts (MoE) management with a batch-1 streaming data approach over a 7.8 billion character corpus. Pre-training large language models typically requires expensive GPU clusters to accommodate massive memory demands for gradients and batch processing. This experiment highlights a promising technique for individual developers to train and align custom models entirely on low-cost consumer hardware. By streaming continuous data in 32K-character chunks with a batch size of 1, the architecture minimizes VRAM usage by eliminating large batch gradient storage. Inactive MoE experts are dynamic and offloaded to disk storage, allowing total model parameters to grow constrained only by available hard drive space.
## BACKGROUND
Continual learning is an artificial intelligence paradigm where models incrementally acquire and accumulate new knowledge over time without erasing previously learned capabilities. Meanwhile, Mixture-of-Experts (MoE) architecture routes inputs to specialized sub-networks, activating only a small subset of total parameters per computational step to keep active compute low.