Anthropic urges AI labs to pause, warns humans risk losing control

AI company Anthropic has urged other leading AI labs to consider a coordinated pause on the development of advanced AI systems. The company warned that rapid technological progress could lead to AI systems improving themselves faster than society can manage the associated risks, potentially resulting in a loss of human control. Anthropic's blog post on Thursday suggested that a temporary slowdown or pause in development would be beneficial. In contrast, rival OpenAI, in a report published Wednesday, argued that democratic governments, rather than private companies, should set the rules and safeguards for AI innovation. Anthropic highlighted the increasing speed of AI in tasks like coding and the potential for "recursive self-improvement," where an AI could design its own successor, which, while offering benefits, also raises control concerns.

AI company Anthropic has proposed a "temporary pause" on AI development and plans to convene policymakers to discuss advanced AI risks. The company detailed progress in its AI model, Claude, towards "recursive self-improvement," where AI could autonomously develop more powerful successors, potentially leading to humans losing control. Experts view this call for a pause and engagement with policymakers as a marketing tactic, noting that Anthropic has previously made similar proposals and that their current AI advancements, while notable in code generation, do not yet demonstrate true recursive self-improvement. This announcement coincides with reports of Anthropic embedding engineers within the NSA for cybersecurity operations, a move some find contradictory to their public stance on AI safety.