Unbounded Labs Releases Bart, a 2.82B Parameter Vintage LLM
Unbounded Labs has released Bart, a 2.82B parameter large language model trained from scratch on 20.1 billion tokens of English text written before 1931. The project aims to investigate whether AI models can generate original scientific ideas without access to modern training data. This experiment tests the limits of AI originality and scientific discovery by isolating the model from modern concepts, helping researchers understand if LLMs can reason independently. Additionally, the release of custom benchmarks and historical datasets provides new tools for studying model generalization and historical reasoning. The model was trained in just five days on a single Nvidia H100 GPU for approximately $807, maintaining a 60% Model Flops Utilization (MFU). The team also released "Vintage CORE", a suite of 20 benchmarks, and a supervised fine-tuning (SFT) dataset containing 416,000 graded Q&A pairs grounded in pre-1930s text.
## BACKGROUND
Training language models on historical data (often referred to as "vintage" LLMs) helps researchers evaluate AI without the risk of benchmark contamination, where models perform well simply because they have memorized test answers from the modern internet. It also allows scientists to explore whether an AI can recreate modern scientific breakthroughs, like general relativity, using only the foundational knowledge available to scientists of that past era.