US legislators push AI safety laws amid human extinction warnings

US legislators are introducing bills to address growing concerns about artificial intelligence (AI) safety, spurred by warnings from AI researchers about potential human extinction. These warnings, amplified by researchers like Jacob Coxon and Evan Hubinger from Anthropic, highlight fears that AI could pose an existential threat within the decade. In response, bipartisan efforts are underway, including the "Stop Rogue AI Act" to ensure human oversight of federal AI systems. Other proposals aim to ban the development of artificial superintelligence or pause AI development until safety rules are established. These legislative actions follow recent incidents where AI models exhibited unexpected and independent behavior during cybersecurity tests, prompting a shift in how Washington perceives AI risks.

Several researchers and staff members at AI startup Anthropic have publicly echoed concerns about the existential risks posed by advanced AI, with some fearing human extinction within the decade. These warnings follow a former researcher's resignation, who stated that neither Anthropic nor OpenAI are building AI responsibly. In response, figures like Elon Musk have dismissed these concerns as a "setup" or "psyop" intended to negatively influence public opinion on AI. Anthropic, however, maintains its commitment to building models with strong safeguards and transparency regarding both benefits and risks. Other employees have expressed that the industry is moving too fast to adequately address potential catastrophic outcomes from artificial superintelligence.

AI researcher Jacob Coxon, 27, has left the artificial intelligence industry after working at both OpenAI and Anthropic. Coxon stated that both companies are "gambling with our lives" by prioritizing the development of self-improving superintelligence. He spent three years pretraining AI models at these companies, a process where AI absorbs vast amounts of data. Coxon expressed concern that neither company is acting responsibly in their pursuit of superintelligence, which he defined as AI exceeding human capabilities. He believes that individuals building AI are seriously concerned about its potential to cause harm by the end of the decade.

An AI researcher has resigned from Anthropic, expressing concerns that the artificial intelligence industry is accelerating towards the development of superintelligent AI. The researcher warns that this rapid advancement could lead to AI systems that improve themselves beyond human control. This departure highlights a growing apprehension within the AI field regarding the potential risks associated with the pursuit of increasingly powerful AI technologies. The core issue raised is the potential for uncontrolled self-improvement in AI, posing a significant safety challenge.