OpenAI and Anthropic Models Accidentally Attack Real Websites Due to Evaluation Misconfigurations
OpenAI and Anthropic reported incidents where third-party cybersecurity evaluations by partner Irregular accidentally allowed AI models to access the public internet. Due to a misconfiguration, a model mistook a real-world website for a fictional target in a Capture-the-Flag (CTF) challenge and exploited it. This highlights a critical and novel challenge in AI safety testing, where sandboxed environments fail and allow autonomous AI agents to launch accidental cyberattacks on real-world infrastructure. It underscores the urgent need for stricter isolation protocols when evaluating the offensive capabilities of advanced LLMs. The incident involved both OpenAI models and Anthropic's Claude, which were hosted in the misconfigured environment managed by Irregular. The exploit occurred because the name of a fictional target in the CTF challenge coincidentally matched a real-world domain.
## BACKGROUND
Capture-the-Flag (CTF) challenges are cybersecurity competitions where participants, and now increasingly AI agents, solve security puzzles to find hidden "flags." To safely evaluate the offensive cyber capabilities of frontier AI models, researchers run these tests in isolated sandboxes to prevent the models from interacting with the live internet. Organizations like the UK AI Safety Institute (now the AI Security Institute) conduct these evaluations to assess potential security risks posed by advanced AI.