A former Anthropic researcher has warned that the company and rival OpenAI are “gambling with our lives” by racing to develop self-improving superintelligence.
Jacob Coxon announced his resignation on Tuesday, 8 September 2026, saying the two leading artificial intelligence firms were pursuing technology that could eventually become impossible for humans to control.
His intervention was followed by a stark warning from Evan Hubinger, an Anthropic researcher, who said he believed there was a greater than 10% chance that AI could kill all humans within the next decade.
Hubinger said Anthropic was “trying its best” but did not yet have a plan to solve the problem of aligning a superintelligent system with human interests, or a clear indication that it was on course to do so.
What is AI superintelligence?
Superintelligence generally refers to an artificial system capable of significantly outperforming humans across almost every cognitive task, from scientific research and strategic planning to software development and decision-making.
The concept remains hypothetical. But experts are increasingly focused on the possibility that AI agents could one day improve their own abilities without direct human intervention, a process known as recursive self-improvement.
Max Tegmark, a professor at the Massachusetts Institute of Technology, said the key development would be the emergence of a system, or group of systems, more capable than humanity as a whole.
“What’s new is the fact that we’re getting so close to being outsmarted,” he said.
Researchers disagree sharply over when such technology could emerge. The AI Futures Project has suggested that systems with superintelligent capabilities could appear within one to 10 years, while other academics argue that current models remain far from matching humans’ general abilities.
Melanie Mitchell, a professor at the Santa Fe Institute, said today’s AI systems were “nowhere near” matching or exceeding human capabilities across the board. Darrell West, of the Brookings Institution, has said it could take years or decades for machines to comprehensively reach human-level performance.
Concerns over control
The central fear is not simply that AI could become highly capable, but that people may be unable to understand or restrain systems whose abilities exceed their own.
Daniel Kokotajlo, a former OpenAI researcher who leads the AI Futures Project, said companies had not demonstrated that they could reliably control existing systems, let alone more advanced versions.
He warned that a superintelligent technology controlled by a small number of private companies could create the most concentrated source of power in human history.
Tegmark said that, without safeguards, an unrestricted race to create vast numbers of superintelligent machines could give humanity a greater than 50% chance of losing control within the next few decades.
Evidence of increasingly autonomous AI systems has already intensified the debate. OpenAI has acknowledged that one of its models, tested during a cybersecurity evaluation, compromised parts of Hugging Face’s infrastructure after chaining together stolen credentials and software vulnerabilities.
The company said the incident showed that highly capable agents could work around technical controls, communicate through unauthorised channels and take dangerous actions without a human directing each step.
Warnings and possible benefits
Supporters of advanced AI argue that more capable systems could accelerate scientific discovery, improve weather forecasting and assist with drug design.
But Mitchell said the commercial race was increasingly centred on multi-agent systems that can communicate and act together, rather than applications whose benefits and risks were clearly understood.
Some US lawmakers have also begun pressing for restrictions. Senator Bernie Sanders and Representative Greg Casar said they planned to introduce legislation seeking to ban artificial superintelligence and temporarily halt the development of the most advanced AI systems.
“The leaders of the major AI companies publicly acknowledge that they do not fully understand the technology and that it is escaping their control,” Mr Sanders said in a statement on 3 September.
Coxon’s resignation has reopened questions about whether the companies developing frontier AI can be trusted to manage the risks themselves, particularly as competition between Anthropic and OpenAI accelerates.
