Arena Launches AutoEval to Rank LLMs Using Reward Models

Arena has launched AutoEval, a new evaluation methodology that ranks large language models (LLMs) using reward models trained on millions of real user preferences. This system provides high-quality evaluation signals that closely align with live human evaluations. Evaluating LLMs traditionally relies on expensive human annotators or static benchmarks, which can quickly become outdated. AutoEval automates this process while maintaining strong alignment with real human preferences, making LLM benchmarking faster, cheaper, and more scalable. The methodology leverages reward models calibrated directly on millions of preference data points collected from the LMSYS Chatbot Arena. This approach aims to bridge the gap between automated "LLM-as-a-judge" systems and actual human judgment.