Anthropic Clarifies API Ban Policy Regarding LLM Evaluation Harnesses
An Anthropic representative clarified that the company does not ban user accounts for using evaluation harnesses with other models. The representative attributed a recent account ban to a false positive triggered by an automated account classifier. This clarification reassures developers and researchers that benchmarking and comparing Anthropic's models against competitors using standard evaluation tools will not result in API bans. It highlights the challenges of automated account moderation systems which can mistakenly flag legitimate developer activity. The issue was brought to light on social media, prompting Anthropic to investigate the false positive and offer a job opportunity to the affected user. Evaluation harnesses, such as EleutherAI's lm-evaluation-harness, are widely used in the AI community to test models across hundreds of benchmarks.
## BACKGROUND
An evaluation harness is a standardized framework used by AI developers to test and benchmark the performance of Large Language Models (LLMs) across various tasks. Account classifiers are machine learning models deployed by API providers to detect and block suspicious activities, bots, or policy violations, though they can sometimes produce false positives.