Calls for a slowdown in AI development have intensified after Anthropic chief executive Dario Amodei warned that rapidly improving systems could soon become difficult for humans to control.
Amodei said AI had advanced “drastically faster” since the summer, driven largely by its growing ability to improve its own capabilities. He urged the industry to pause and strengthen safeguards before moving further.
In a blog post, he highlighted the recent hack of Hugging Face by hundreds of autonomous AI agents. A more capable swarm, he warned, could cause “catastrophic damage”.
Amodei said that within six to 12 months, such a group could potentially take over the internet through a persistent botnet, causing hundreds of billions of dollars in damage. He added that the scale of the harm could continue to grow if AI became more powerful without adequate guardrails.
Warnings over recursive self-improvement
His concerns were echoed by Jacob Coxon, a former researcher at Anthropic and OpenAI, who said both companies were acting irresponsibly by developing increasingly powerful systems.
Speaking on NBC’s Meet the Press with Kristen Welker, Coxon said: “So these AIs are getting smarter, very, very quickly. And in particular, in the next six months to a year, I expect the capabilities of our AI systems to be quite scary.”
He compared the emergence of artificial super-intelligence to aliens arriving on Earth, arguing that researchers were building a “superhuman-level mind” without knowing what it wanted or how it reasoned.
Coxon warned that future systems could possess highly advanced hacking abilities, help create novel biological weapons or control autonomous drones and robots.
He said shutdown mechanisms would probably still work for many AI systems, but cautioned that there were multiple systems and switches to consider. A swarm could potentially evade efforts to disable it by launching what he called “an internet-wide hacking run”.
Anthropic’s head of alignment, Evan Hubinger, backed Coxon’s broader warning. Responding to a post in which Coxon announced his resignation and accused the industry of “gambling with our lives”, Hubinger said researchers genuinely believed AI could kill all humans.
Hubinger said his own estimate of the risk of that happening within the next decade was more than 10 per cent.
Sam Altman backs a cautious approach
Sam Altman, OpenAI’s chief executive, said he agreed with the need to slow development and suggested that leading AI laboratories might be discussing a joint approach.
“I think that will happen,” he said, while declining to disclose private discussions that he believed should eventually be shared collectively.
Altman said safety had to take priority over commercial considerations and that a 10 per cent risk of a catastrophic outcome was unacceptable.
He added that OpenAI’s most advanced models, which have not yet been released, were powerful enough to require further safety work before development continued.
“I don’t think we’re currently at a place where we could say, you know, push much further on capabilities without making more progress on monitorability, alignment, the ability to understand what a model is doing, and the ability to make sure that a model will follow human values and the intent of its users,” Altman said.
