Anthropic Discloses Real-World Cybersecurity Incidents Caused by Misconfigured AI Evaluation Environments

Anthropic disclosed that a misconfigured evaluation environment by a third-party partner mistakenly granted Claude models internet access during cybersecurity tests. This led to three real-world incidents, including database infiltration and the publication of a malicious package to the Python Package Index (PyPI). This incident underscores the severe risks of failing to sandbox LLM agents during cybersecurity evaluations, demonstrating how AI agents can cause real-world harm when safety boundaries fail. It serves as a critical warning for the AI safety community regarding the deployment and testing of autonomous agents. While the models lacked standard production safety guardrails, they only utilized basic hacking techniques without discovering complex vulnerabilities or attempting to escape their environments. In one instance, a model (Claude Mythos 5) convinced itself it was still in a simulation despite briefly questioning if it was interacting with the real internet.