The White House Accord on Superintelligence has placed AI biosecurity risks at the centre of the safety debate, but critics warn that its voluntary safeguards do not define when companies must act or what should trigger a wider response.
President Trump and leaders of some of the largest AI companies signed the agreement on Tuesday. It calls for internal monitoring, independent assessments and board oversight of risks including the possible misuse of AI in biological research.
However, the accord leaves companies largely responsible for deciding what those protections should involve and when a threat has become serious enough to warrant action.
Bill Gates has warned that AI crossed several important thresholds this year, including in its potential to support a biological attack. He identified the most concerning test as whether an AI system could design a bioterrorism weapon capable of killing millions.
Last week, the chief executive of Anthropic called on the United Nations Security Council to support a ban on using AI to create biological weapons. The appeal followed the company’s disclosure of five cases of biological misuse involving its models, including a researcher examining how H5N1 influenza adapts to mammals and causes illness beyond the respiratory tract.
Anthropic banned the accounts involved. H5N1 does not currently spread efficiently between people, but there is little population immunity and roughly half of recorded human cases are fatal.
From digital designs to physical capabilities
The central concern is not simply whether an AI model can answer a dangerous biological question. It is whether it can help someone without advanced expertise overcome experimental problems that previously required specialist judgement.
A model capable of producing a research protocol is not, by itself, a person capable of creating a pathogen. Turning a digital design into a physical threat would still require access to synthesised DNA, equipment and a laboratory in which to grow a virus.
But the barriers between those stages could weaken as AI systems improve. The key warning signs would include a concerning AI-generated sequence being ordered from a DNA synthesis provider or used in an automated laboratory, as well as evidence that dangerous biological work requires substantially less hands-on experience.
The proposed safeguards include independent red-team testing of AI models before deployment, alongside watermarking and tracing systems for biological design tools. Screening would also need to be strengthened at DNA synthesis companies, cloud laboratories and manufacturers of benchtop synthesizers.
A bipartisan Senate bill on expanding such screening has not moved since January, while the Bipartisan Commission on Biodefense has recommended watermarking and tracing tools for biological design systems.
The argument for setting these thresholds in advance comes from an earlier H5N1 incident. In 2023, officials were told that a father and his daughter in a remote Cambodian village had been infected, with the daughter dying.
The immediate question was whether the cases marked the beginning of sustained human-to-human transmission and a pandemic that could kill millions. The decision was taken not to brief President Biden immediately, because surveillance data from nearby emergency departments did not show an unexpected increase in cases.
Officials had already agreed that sustained transmission between people would be the trigger for escalation. That approach avoided both dismissing a genuine emergency and responding to an isolated incident as though it were the start of a pandemic.
The same trigger-and-action approach has not yet been applied consistently to AI and biology. The warning is that “not yet” may describe the current position, but it is not a plan for deciding what happens when the barrier between AI-generated knowledge and physical biological capability begins to erode.
