GPT-X3-150M Claims Top Spot on Open SLM Leaderboard
GPT-X3-150M has claimed the number one position on the Open SLM Leaderboard, surpassing HuggingFace's SmolLM2-135M model. This milestone was achieved by adopting a novel GDN2 and GQA hybrid architecture. This development highlights the rapid progress and efficiency gains in small language models, making advanced local AI capabilities more accessible for consumer hardware. Such architectural innovations push the boundaries of what ultra-compact models can achieve. The model utilizes a hybrid approach combining Gated DeltaNet-2 (GDN2) state space layers and Grouped-Query Attention (GQA). At only 150 million parameters, it outperforms previous market leaders in the ultra-small language model category.
## BACKGROUND
Small language models (SLMs) are compact AI models designed to run efficiently on resource-constrained devices like smartphones or edge hardware. Architectures like GQA help reduce memory bandwidth bottlenecks during inference, while newer state space layers like GDN2 aim to improve data efficiency and long-context processing.