AI Model Usage Costs Vary Greatly Across Languages Due to Token Disparities
Recent research reveals that AI models consume vastly different amounts of tokens depending on the language used, creating hidden cost disparities of up to 17 times between high-resource and low-resource languages. This disparity imposes a hidden 'tokenizer tax' on users of non-English or low-resource languages, leading to higher costs, slower response times, and limited context window capacity that hinders equitable participation in the AI economy. The study found that token consumption can differ by 1.5 to 5 times even among the top 20 internet languages, with the gap widening up to 17 times across a broader range of 200 languages depending on the model and script used.
## BACKGROUND
Large language models (LLMs) break down text into smaller units called tokens using tokenizers before processing them. Because tokenizers are typically optimized for data-rich languages like English, they break less-resourced languages into a much higher number of fragments, inflating processing requirements.