Audit of Major LLM Benchmarks Reveals 12% of Questions Were Broken

An independent audit of prominent LLM benchmarks—GPQA, MMLU-Pro, and MMMU-Pro—revealed that up to 12% of their questions were broken due to formatting errors, incorrect answer keys, or multiple valid answers. In response, the researcher released cleaned versions of these datasets, which saw top-tier models' accuracy jump to around 98%. This audit explains why top AI models seemed to hit a performance ceiling of 92-93% on benchmarks like GPQA-Diamond, showing that the limitation was due to dataset quality rather than model capabilities. By providing cleaned benchmarks, researchers can now more accurately measure the reasoning limits of frontier models without noise from faulty questions. The audit covered GPQA (Diamond and Extended), MMLU-Pro, and MMMU-Pro, and the author has shared the cleaned datasets on Hugging Face along with integration for the EleutherAI lm-evaluation-harness. A detailed ledger documenting why each broken question was flagged has also been made publicly available.