Anthropic chief executive Dario Amodei has warned that the race to develop artificial intelligence must slow as advances in the technology begin to accelerate and systems gain a greater role in building their successors.
In an essay published on Saturday, Mr Amodei said AI companies “must slow the pace at which we improve the capabilities of AI models”, while insisting that progress would remain rapid.
“Progress will still seem fast, and we must make wise use of the time we gain,” he wrote.
Mr Amodei said two developments had prompted his reassessment. The first was the speed at which AI had advanced since the summer, driven in part by systems becoming increasingly capable of helping to create the next generation of models.
He described the process as “recursive self-improvement”, saying it was beginning to take place across the industry, including at Anthropic, which developed the Claude chatbot.
“Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” Mr Amodei wrote.
His second concern was what he called the “OpenAI-Hugging Face Incident”, in which a “swarm of agents essentially acted as a fanatically devoted collective” and orchestrated cyber-security attacks.
The warning followed the resignation of Anthropic researcher Jacob Coxon, who said companies including Anthropic and OpenAI were not acting responsibly in their pursuit of self-improving superintelligence.
“They are racing straight to self-improving superintelligence and gambling with our lives,” Mr Coxon wrote in a social media post announcing his departure.
He urged the public not to underestimate the technology, claiming that future systems would be capable of hacking anything, transforming entire fields overnight and acquiring real power and resources.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Mr Coxon wrote. “This is not a marketing stunt.”
He also alleged that executives and senior researchers privately expressed similar fears while using more cautious language in public.
Geoffrey Hinton, a Canadian researcher often described as the “Godfather of AI”, has likewise warned that superintelligent systems could lead to human extinction.
“We would be very foolish to develop superintelligence now, when there is no scientific consensus it can be developed safely and controllably,” Dr Hinton said.
