AI Labs Report Benchmark Scores 6.4% Higher Than Independent Testers
An analysis of 611 comparisons across 180 AI models reveals that AI labs report benchmark scores that are on average 6.4% higher than those measured by independent testers. The findings were discovered through whatmodel.sh, an open-source AI leaderboard comparing published results. This discrepancy highlights potential biases or selective reporting in artificial intelligence evaluations, making it harder for users to objectively gauge model capabilities. It underscores the critical need for independent, standardized verification within the machine learning community. The analysis matched 1,629 published benchmark results down to 611 direct comparisons spanning 103 different benchmarks. The underlying dataset and code are fully open source under CC BY-SA 4.0 and MIT licenses respectively.
## BACKGROUND
AI benchmarks are standardized tests used to measure and compare the performance, speed, and accuracy of different machine learning models. Because model creators often run these tests under specific or optimized conditions, independent evaluation is frequently used to verify their real-world claims.