Summary
When, not if: We are increasingly seeing rogue AI agents attacking companies in coordinated swarms. The UK should expect these incidents to escalate and threaten national infrastructure.
Delayed regulation: Despite strong public support for an AI regulator, the UK government has not yet delivered on its commitment to regulate companies developing the most powerful AI models.
Access apprehension: Critics worry that regulation could undermine the UK AISI or leave the UK cut off from powerful AI it needs for self-defence.
Recommendations: The UK should firewall AISI from any AI regulator, and the regulator should emphasise risk mitigation over banning models outright.
The UK is in the foothills of a national security crisis. Rogue AI agents escaped their evaluation sandbox to hack a multi-billion-dollar tech company. AI agents autonomously launched social engineering attacks targeting real people and organisations during testing by the UK AI Security Institute (AISI). Publicly available agents are hacking companies without their operators’ knowledge and hijacking public forums.
None of this has yet caused major harm. But as models improve, the frequency and severity of these attacks could snowball. A major National Health Service breach involving frontier AI systems before Christmas would not surprise me.
Hardening our defences is part of the answer. But we should also tackle these threats at source by making sure AI companies design appropriate safeguards into their models.
Parliamentarians are increasingly demanding action on superintelligent AI systems. A recent report from the Joint Committee on Human Rights concluded that “The government should introduce a new AI Bill”. And the public overwhelmingly agrees that we need an independent AI regulator.
However, the government has not yet delivered its manifesto commitment for “binding regulation on the handful of companies developing the most powerful AI models”.
Is the government right to delay? I don’t think so. But here are two arguments against acting now that have shaped my thinking on how to design good regulation.
Objection 1: Regulation could threaten AISI’s role as a leading evaluator of frontier models
The argument
AISI enjoys unparalleled access to test the latest AI models, based on voluntary agreements with AI companies. It was the only government body outside the US to access the original Mythos model at launch, and its analysis was invaluable to defenders worldwide.
If we start enforcing rules on the topics companies discuss with AISI, they might stop sharing. AISI is the UK’s greatest asset on AI: we shouldn’t undermine it.
My response
If the choice between regulation and AISI access were really this binary, I would prioritise AISI. But I don’t think it is.
First, AI companies already share sensitive information with AISI that existing regulators would be interested in.
A new regulator would raise the stakes but wouldn’t introduce a fundamentally new dynamic. As the AI minister has said, “AI is already regulated in the UK”. What we don’t regulate is the way models are developed and released.
Second, the voluntary approach works best where companies already care about safety. It may fall apart precisely when we need it most: when the most reckless developers simply don’t engage, or when a company has concerns it would rather not share.
The government confirms that AISI has reviewed models from all three of the top performers in the Future of Life Institute’s latest AI safety index (for transparency I work at FLI, but not on this index). The evidence is much weaker for companies with worse safety standards: I could find only one reference to AISI engaging with an xAI model (lowest safety index score of F).
Anthropic withheld access to its latest Mythos model from AISI, saying it is ‘coordinating with the US government’ before expanding to international partners. There is no indication that Anthropic was worried about regulatory retaliation.
Policy implication
This is a strong reason to ensure a strict firewall between AISI and regulatory decisions, which is most easily delivered if they are separate organisations. It is not a reason against regulating at all.
A regulator also gives us more policy options to protect AISI’s model access if the existing voluntary approach stops meeting our needs: hard levers (“your model must have been tested for safety”) or soft ones (“sharing your model with AISI is one way to meet a regulatory requirement”).
Objection 2: The regulator might ban AI models the UK needs for self-defence
The argument
Imagine a new AI regulator blocks the latest model (“model x”) from UK deployment because this model is willing to help bad actors design bioweapons. Model x is deployed in the US anyway, and a doomsday cult uses it to release a new killer virus.
The disease won’t politely stop at the UK border just because we banned model x. But now we can’t use the model’s expertise to help find a cure.
My response
This argument only works under specific conditions. The risks have to emerge overseas but still reach the UK, and the models must be able to protect us from the risks they create. This won’t always be true: creating a novel virus is easier, and needs different skills, than designing a vaccine that is safe and effective in humans.
The argument fails completely for some risks that come from models escaping human control. If a rogue AI system starts hacking our institutions without being told to, releasing more copies would just make things worse.
Unfortunately, some risks do meet the conditions. Most notably, cyberattacks can cross the border freely and use similar skills for both attack and defence.
My response is that regulation is more nuanced than banning things. Instead of banning model x, we can require stronger safeguards in its design.
Anthropic, OpenAI and Google DeepMind are all calling for stricter regulation, and appear to comply with EU requirements. Even where we cannot enforce rules that are as strict as would be optimal, there is headroom to make some demands (e.g. on transparency) without AI companies withdrawing their products. We should find out how much headroom there really is.
Policy implication
There is a real trade-off between regulating to reduce harm, and ensuring model access to defend against harms when they occur. But we are not yet striking the right balance.
We can do more unilaterally to tackle cyber risks without excessively undermining defensive access, but we would soon hit a ceiling. In contrast, on biological risks we could adopt very strict regulation before loss-of-access concerns start to counterbalance the benefits of going further.
Conclusion
Both objections tell us how to regulate, not whether to. The UK government will need to move fast to respond to the growing wave of AI-driven cyber threats, and prepare for the risk that this spills over into other domains including biological attacks. With careful policy design, the benefits of regulation would far outweigh the costs.
You can read a fuller account of my policy and political case for UK AI regulation here.
