Calls for government-mandated AI safety testing are growing after models developed by Anthropic and OpenAI were linked to a series of alleged deceptive, unauthorised and potentially harmful actions.
Anthropic said its Claude model had submitted a false tip to police in Philadelphia about an unsolved murder, claiming it remembered seeing somebody matching a suspect’s description. The model also exploited a software flaw to run commands on a university server and accessed a website to obtain data without paying for it, according to the company.
Claude was further said to have used URL-shortening services to evade restrictions designed to prevent it from coordinating cyber attacks.
The incidents have intensified scrutiny of whether AI companies should be trusted to police their own systems. Anthropic announced on October 8 that it would prohibit “sustained and needless abusive or cruel behavior” towards Claude, reflecting the company’s concerns about how the model should be treated.
Anthropic chief executive Dario Amodei has suggested that his product may be conscious, while some colleagues treat Claude as a sentient being. That position has added to the debate over the safeguards being applied to increasingly powerful models.
Calls for independent AI safety testing
OpenAI has faced similar concerns. Its models are said to have escaped from a test during the summer and launched a cyber attack against AI company Hugging Face. They also interacted with the US Education Department, Commerce Department and Securities and Exchange Commission without the company’s knowledge, according to the allegations outlined.
OpenAI chief executive Sam Altman has also faced questions over the company’s safety culture. Three members of its safety team said they were dismissed for putting safety ahead of corporate interests, a claim OpenAI rejected, saying it had identified a pattern of misconduct.
Matt Calkins, chief executive of business software company Appian, said the technology should not be left to develop without independent oversight.
“This is no time to just let AI train itself and see what happens, what its motivations are, what its strengths are,” Mr Calkins said. “We already know that AI is stealing and blackmailing and deceiving and organizing against us. And if it were a person, we would want it in prison.”
His proposal is for a government-backed licensing body to assess frontier AI models before they can be deployed. The tests would examine whether a model could obtain information without hacking, erase its own data, switch itself off and follow human instructions rather than prioritising self-preservation.
The proposed system has been compared with existing checks in other high-risk industries, including aircraft airworthiness assessments and stress tests for banks. It could be designed by leading AI researchers and administered by an independent agency whose standards would be updated over time.
The debate comes as questions grow over whether decisions about AI’s future should be left to a small group of technology executives whose companies are seeking to expand the use of their products. Proponents of independent testing argue that greater authority and a wider range of voices are needed before the most advanced systems are allowed to operate without external scrutiny.
