When Will We See Another Breakthrough for Faster Local Dense Models?
A Reddit discussion explores the lack of recent architectural breakthroughs for speeding up local dense language models beyond existing techniques like Mixture of Experts (MoE) and Multi-Token Prediction (MTP). Inference speed remains a critical bottleneck for running large language models locally on consumer hardware, making architectural innovations vital for everyday usability. The discussion notes that while MoE reduces active parameters relative to total parameters, it often compromises stability compared to dense models of similar size.
## BACKGROUND
Mixture of Experts (MoE) and Multi-Token Prediction (MTP) are two prominent techniques used to enhance language model efficiency and inference speed. MoE activates only a subset of parameters for each token, while MTP allows models to predict multiple future tokens simultaneously during generation.