Developer Proposes Custom 9.4B Dense LLM Combining Engram, AttnRes, and NoPE Layering
An independent open-source developer introduced a custom 9.4B dense LLM architecture designed for single-GPU training, seeking community feedback before executing the pre-instruction-tuning phase. The model integrates DeepSeek's Engram memory tables, Moonshot AI's Attention Residuals (AttnRes), a 3:1 RoPE to NoPE layer ratio, and knowledge distillation from Llama 3. This project highlights how solo researchers can combine state-of-the-art architectural innovations on accessible consumer hardware through knowledge distillation and code optimization. If completed, it could provide the local AI community with an open-source 9B-class model that pushes structural design boundaries beyond standard Transformer baselines. The author optimized the training pipeline specifically for a single RTX 6000 Pro GPU, utilizing logit-level data extraction from a Llama 3 teacher model to streamline pre-training. All model weights, training datasets, and code will be fully open-sourced, and the developer also submitted fixes to vLLM that offer up to 10x-100x prompt loading improvements.
## BACKGROUND
Modern language models increasingly adopt structural modifications beyond traditional Transformers. DeepSeek's Engram module adds static N-gram memory lookups to offload knowledge lookup from compute layers, while Moonshot AI's Attention Residuals (AttnRes) replace fixed residual shortcuts with learned depth-wise attention. Additionally, hybrid RoPE/NoPE designs alternate layers with positional embeddings (RoPE) and non-positional layers (NoPE) to improve long-context handling and reduce recency bias.