Anthropic Backs AI Safety Petition Citing Recursive Self-Improvement Risks
Anthropic has announced its support for an AI safety petition signed by its CEO, co-founders, and senior staff. The company emphasizes the need to deliberately pace frontier AI development, referencing their recent research on recursive self-improvement. As AI systems approach the ability to autonomously design and build their own successors, the risk of an uncontrolled intelligence explosion increases. This petition highlights a growing consensus among leading AI labs that governance frameworks must adapt to manage the transition to artificial general intelligence (AGI). Anthropic's research warns that AI may soon begin recursive self-improvement, transitioning from current AI-assisted research to fully autonomous improvement cycles. However, current evidence suggests open-ended self-improvement remains bounded by compute constraints, grounding requirements, and collapse dynamics.
## BACKGROUND
Recursive self-improvement (RSI) is a process where an AI system rewrites its own code or designs its successors, potentially leading to rapid capability gains. While bounded self-refinement is already common in the industry, open-ended RSI raises significant safety concerns as systems could surpass human control. Anthropic's research outlines various scenarios of this technology, urging society to prepare for these autonomous feedback loops.