Developer Builds Experimental 2B LLM with 1B Engram Table and OLMo Tokenizer

An independent developer shared an update on training an open-source, Apache 2.0-compliant 2B-parameter LLM integrated with a 1B Engram memory lookup table. By combining an OLMo tokenizer with soft-label probability distillation from a 7B model, the custom architecture achieved initial language coherence after just 15 million tokens of training. The project demonstrates how independent builders can bypass restrictive licensing frameworks like LLaMA by leveraging fully open components such as Ai2's OLMo. It also explores innovative hybrid architectures that allow deep language models to be trained on standard consumer GPUs with 24GB of VRAM. The model shrinks `d_model` to 2048 to accommodate 40 sliding window and global attention blocks while allocating 50% of its total footprint to the static Engram N-gram table. It preserves around 65% of the source embedding content through down-projection and avoids common early-training repetition loops by learning soft probability distributions.