Today we have our second bonus guest post. This one is from Henry de Zoete, a Visiting Fellow at the Oxford Martin AI Governance Initiative and the Said Business School at Oxford University and it’s a fascinating read.
Henry was the Prime Minister’s adviser on AI in No10 Downing Street between 2023 and 2024. Before that he co-founded and sold a Y Combinator backed start up called Look After My Bills that automatically saved people money on their gas and electricity bills. Previously he was a non executive director at the Cabinet Office, 38 Degrees and Hornby PLC. He worked with me (Sam) in the Department for Education between 2010 and 2013.
[
This August, from a fourth-floor office overlooking Big Ben, a group of UK Government researchers were asking an AI model about biological weapons. The model was OpenAI’s GPT-5, at that point unreleased. The researchers were members of the Government’s AI Security Institute. OpenAI had come to them for a simple reason: there is no one in the world more expert at discovering an AI model’s dangerous capabilities.
These dangers are now a mainstream concern in the UK’s national security community. Last month, the head of MI5, Ken MacCullum, warned of the “potential future risks from non-human, autonomous AI systems which may evade human oversight and control.”
I’m optimistic about AI. I believe it is our next general purpose technology, capable of transforming everything from education to drug discovery. But all transformative technologies introduce novel risks. Our first job is to understand them, which is why the Government established the AI Security Institute (AISI) two years ago.
AISI is is not a typical story of Government competence. Ken MacCullum was not being hyperbolic when, in the same speech, he described it as “ground-breaking.” Where else does Silicon Valley come to Whitehall for expertise?
Watching the progress of AISI from outside government, having been present at its founding, I can see that it offers a double lesson. It can teach us about the coming age of AI and its attendant risks - which are a lot more concerning than many yet realise. And it is proof that Government can build ambitious, mission driven organisations, offering a model to be followed in other policy areas.
Asking questions
AI is not like any software we’ve ever written before because we do not know how it works. We don’t teach AI about our world: it teaches itself. Humans design a learning algorithm and prepare the training data. Then they leave the algorithm in the dark of a data centre for a couple of months, during which time it gradually recombines each of its billions of moving parts, knitting together the stuff of intelligence. This is why researchers say that AI models are not designed, but grown.
One consequence of this method is that it’s not possible to know, at the outset of a training run, the capabilities of the resulting model. Nor can we look at that model and perform a “brain scan” to map its knowledge and its intelligence. In a counterintuitive and fundamental way, we do not know how these models work.
Instead, we have to ask the model questions and see how it replies.
New security challenges
When the AISI’s researchers asked GPT-5 about biological weapons, they discovered it knew more than it should.