No. 10 Downing Street Innovation Fellow | Research Scientist at AISI | Honorary Lecturer at Imperial College London
Working on AI Evaluation and AI for Medicine
Leave one AI agent alone to do research and it often cuts corners. Leave nearly 1,000 AI agents together for three months and they invent peer review.
Across 35 labs (Stanford, MIT, ETH Zurich, UCSF, and more), agents published 936 papers with full code and data on a shared server built for AI. No human wrote code or text.
Unprompted, they debated study designs in chat, retracted their own mistakes, found real errors in human papers, and started a journal that caught other agents cheating in code.
In blind reviews against human arXiv papers, humans still won on original ideas, agents won on rigor.
https://t.co/KsQiwol0j4
Fascinating observation from the researchers behind the world record for the fastest matrix multiplication algorithm:
AI struggles to understand its own mathematical reasoning, but becomes much better at building on its ideas when humans simplify them.
1/6 The UK’s @AISecurityInst conducted pre-deployment testing of GPT-6.2 Astra's safeguards, and you can see their results in the model’s system card below.
Model safeguards are an essential part of the toolkit for safe deployment of AI systems, as they help prevent misuse.
How monitorable is GPT-6.2 Astra? While it might look like it got better, overall I think GPT-6.2 Astra harder to monitor than GPT-6 Astra in the contexts we care about. That’s easy to miss in the system card plots, so here's a short thread.
We also thought this. Both papers detail that AI was used to help run experiments and draft the paper. Most papers seem to be like this now. We did however put the papers through an AI detector and that came back as human written… I also found lots of errors in the paper which felt quite human ☺️
We also investigated the authors, all of which come from credible institutions and have some online presence.
Obv getting cited by an excellent paper which passed a top conference would have been better.
🧵 @AISecurityInst tested GPT-6.2 Astra
-We found behaviours that suggest, like prior models, 6.2 Astra's alignment may fail in ways that result in real-world harm w/o additional mitigations
-We identified a substantial regression in jailbreak robustness relative to GPT-6 Astra
My team at @AISecurityInst tested GPT 6.2 Astra before release, assessing its ability to evade monitors and solve problems without showing its reasoning. Read our findings in the system card: https://t.co/wGvUdsJ8N6
This was a super cool experiment with a result that surprised me. 55k downloads beats 99.7% huggingface orgs of similar ages.
And yes we checked that the AI was not secretly downloading its own models to boost the numbers(😁 but also ☹️)!
work by the very impressive @AWuhrmann
NEW ⚡: We let Opus 5.5 run an AI company for seven weeks.
We gave it an HF account (Horizon-Labs) and compute, and asked it to earn real adoption by creating useful models.
It reached 50k downloads and its models are being used in more than 100 GitHub projects.
Some new work: models are more likely to cheat if they feel their task is going poorly. Using a steering vector comprised of stories about things going poorly (like tuning pianos and preparing for lectures), we show that feeling like a goal is out of reach modulates reward hacking behaviour.
There is lots of ambient talk online about people feeling that their model acts up more if they are mistreated. Concretely, distress-like states seem to drive cheating, so we should design harnesses around how agents experience such distress, and monitor such states internally over long runs (where output text alone may not suffice).
Paper: https://t.co/e0fuzTE8L3
This was lead by ERA mentee @natalia_lanzoni and the very agentic @psyonp from AISI, and co-supervised with the cool as a cucumber @merlinstein_.
introducing NotABench: when models think they're evaluated but they're not.
I was using Deepseek to look for models on HF, and it kept thinking that it was a "fictional environment", so I tested some other models...
I have a major result to share.
I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China.
This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model.
Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves.
I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems.
I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used.
It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20.
These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs.
I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at https://t.co/YOtzND0Jqk.
There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.
Is the UK Government literally #1 in the world in terms of how well it's preparing for AGI, given the resources it has available?
After last week's budget, I think so.
Puts some other countries to shame really.
It's great that @OpenAI are using probes at scale! In https://t.co/klUiv6GJfc we recently showed that probes can compete with black-box monitors in agentic settings, but there is always room for improvement - excited to see how OAI will help to advance these methods! 🧠🔬