Developer builds a 250M parameter LLM from scratch that runs on CPU in 60MB
A developer has trained a custom 250M parameter LLM (SHADOW-250M) from scratch on 30 billion tokens of the FineWeb dataset. The model is quantized to sub-2-bit, allowing it to run at 400 tokens per second on a standard laptop CPU with a deployment size of just 60 MB. This project demonstrates the feasibility of running highly compressed, lightweight LLMs on consumer-grade CPU hardware without relying on heavy frameworks or GPUs. It also showcases an innovative long-context retrieval mechanism (up to 100 million tokens) using disk-offloading, which could inspire new approaches for edge AI applications. The model uses a unique vocabulary representation where every token is a fixed 512-bit code rather than a traditional embedding table, eliminating trained embedding parameters. For long contexts, the most recent 2,048 tokens are kept in a standard FP16 KV cache, while older history is compressed to 1 bit and offloaded to disk.
## BACKGROUND
Quantization is a technique used to compress neural networks by reducing the precision of their weights, with sub-2-bit quantization representing an extreme form of compression that typically degrades model accuracy. A Key-Value (KV) cache is commonly used in LLMs to store previously calculated attention vectors, speeding up generation but consuming significant memory for long sequences. FineWeb is a high-quality, web-scale English dataset curated by Hugging Face for pretraining large language models.