Meta Trains Small AI Models on Raw Bytes Instead of Tokens

Meta researchers trained small 1B-parameter language models to process raw bytes instead of traditional tokens using a distillation approach from a larger model. Their scaling projections suggest that while token models perform better initially, byte-level models continue improving and could eventually outperform existing small models. This research explores a fundamental architectural shift that could eliminate traditional tokenization bottlenecks and language biases in large language models. Bypassing tokenizers may resolve common issues like spelling errors and miscounted letters while supporting diverse scripts more natively. The byte-level models required about six times less training text to reach equivalent levels, but processing raw bytes increases computational steps significantly since a token typically spans four to five characters. Furthermore, the performance projections surpassing Llama and Gemma are currently theoretical predictions based on extended trend lines rather than tested training runs.