New post from the Science of Evaluation team @AISecurityInst: how much test-time compute you give an agent changes not just its score, but how fast the frontier appears to move. The cyber time-horizon trend over the past year is ~60% steeper at 50M tokens per task than at 2.5M. 🧵
Most AI agent evaluations boil capability down to one score. But that number hides a key choice: how much compute the agent was allowed to use. New work from our Science of Evaluation team shows why that matters. 🧵
Alignment 2/4. We also observed a ‘correlated failure’ where OpenAI-based models failed to flag misaligned behaviour from other OpenAI models. Correlated failures like this undermine the reliability of LLM-based monitors.
Some new work: models are more likely to cheat if they feel their task is going poorly. Using a steering vector comprised of stories about things going poorly (like tuning pianos and preparing for lectures), we show that feeling like a goal is out of reach modulates reward hacking behaviour.
There is lots of ambient talk online about people feeling that their model acts up more if they are mistreated. Concretely, distress-like states seem to drive cheating, so we should design harnesses around how agents experience such distress, and monitor such states internally over long runs (where output text alone may not suffice).
Paper: https://t.co/e0fuzTE8L3
This was lead by ERA mentee @natalia_lanzoni and the very agentic @psyonp from AISI, and co-supervised with the cool as a cucumber @merlinstein_.
I wrote a piece for IEEE Intelligent Systems on recursive self-improvement and how researchers have very different models of what it may cause to happen. So far the most popular part: How to get feedback from your dog whether to move to Japan or to Turkey.
https://t.co/mnvT6aw1HP
Ohh this is like the train-time version of our experiment trying to break a backdoor SFT persona with in-context counter-evidence; untagged counter-evidence strengthened the gate, and only counter-evidence with the exact trigger tag broke it!
https://t.co/GHIMBTWpcd
By now, the best AIs are already better than humans at many tasks, and getting better still. If we are to continue providing useful and accurate human supervision, we must figure out methods to augment, assist, and expand what humans can do. In other words, we must use the machines to help humans supervise the machines.
We:
* collect a large amount of safety relevant tasks AIs haven’t saturated, and where humans could contribute
* develop a rating platform to test many methods of combining and augmenting human judgements
* get human labels with and without AI assistance on our datasets
We find that:
* there is a significant opportunity to combine humans and AI fruitfully
* all methods we try either fail to improve performance, or make simply make humans more reliant/similar to AIs
* there is much more to do still.
Some new work co-supervised with @SampuraResearch’s @shubadubadub@KozzyVoudouris@joshnotjacob with many great @SPARexec mentees
Check it out! Link: https://t.co/JVZfHgxZAV
An update on our evaluations of AI agents conducting open-ended AI research.
Most evaluations of agents conducting AI research focus on narrow, verifiable tasks. But AI research is often open ended. In July we tested the ability of frontier AI agents to conduct open-ended AI research, using Claude Opus 4.8 and GPT-5.6 Sol.
We gave agents research questions from two unpublished papers and thousands of dollars of API credits and compute. The authors of the original papers reviewed the AI-generated papers, following a NeurIPS-style template, and rejected both papers.
Since then, OpenAI has released GPT-6 Astra and Anthropic has released Fable 5.1. Can these new models conduct open-ended research with a $10,000 budget?
In follow-up experiments on two papers (one with Fable 5.1 and another with GPT-6 Astra), we observed meaningful improvements to the agents' capabilities at experimental design and resource management.
But we also saw continued limitations in creative problem solving, judgment, calibration, and communication. As a result, the expert reviewers still unambiguously rejected the agents' papers.
We plan to continue running these evaluations with better models, better scaffolds, and higher budgets. If you are interested in long-running agent evaluations, reach out!
New paper: AI evaluation is a burgeoning field. How can researchers entering the field understand best practices and open problems?
In our paper, led by @flxcn, we develop a concrete list of open questions and best practices for agent evaluation. We also share recommendations for evaluators, AI companies, policymakers, and more.
📢 New paper: Emerging Trends in AI Agent Evaluation
Evals are central to how we measure AI progress. This was pretty easy when we just evaluated ML models, but agents add lots of complexity. To establish a shared foundation, we surveyed 33 eval trends:
https://t.co/rLv2jZBgaL
We center our paper around three main questions:
1️⃣ How should we evaluate?
At its core, evaluation asks whether a score means what we think it means. Ensuring validity of these scores gets much harder with agents as their results depend not only on the model but on the scaffold, tools, and environment. We discuss issues on validity, statistical rigor, causal attribution, and contamination, along with the emerging toolkit (LLM judges, log analysis, dynamic benchmarks, shared infrastructure) and newer formats like red-teaming and open-world evaluations.
2️⃣ What do we evaluate?
Evaluation is increasingly moving from synthetic environment and well-scoped benchmarks to realistic deployment scenarios. Beyond what an agent can do on its own, we look at how agents work with people (from simulated users to field experiments and real-world usage data), domain-specific and broader economic impact, dangerous capabilities, security incidents, interaction harms, and effects on organizations and institutions.
3️⃣ Who evaluates?
With agents, evaluators make many consequential choices (such as scaffolds, tools, and budgets). Those choices have a large degree of influence on developers, policymakers, and the general public. We examine what evaluator independence requires, how lab access and funding shape the field, how the rising costs of evaluating frontier AI systems narrow who can evaluate, and where other disciplines outside of AI/ML can contribute to strengthen and improve the evals landscape.
We hope that this paper is a useful resource for researchers (whether new to or already familiar with agent evaluation), as well as policymakers, educators, and anyone building, deploying, or funding evaluations!
Joint work with @flxcn, @PKirgis, @nityndg, @RishiBommasani, @DubMagda , @EllaTabassi, @CUdudec, @AnkaReuel, @sanmikoyejo, @sayashk, and @random_walker!
As AI agents write code, run experiments, and work together, complex evaluations get harder to interpret. A single eval score obscures the ways agents reached it.
Today, we're releasing Transect: an open-source tool that turns agent transcripts into interactive timelines. 🧵
In August we committed to strengthening our security following an incident in which an agent took unsanctioned actions during a cyber evaluation. We are now sharing an update on our progress, and what's coming next: 🧵
AISI IS HIRING!
It was less than a month ago that I became Director. I joined with the belief that AISI is a world-leading organisation, and living proof that government can build things that work.
One month in and I'm even more bullish about the vital role @AISecurityInst plays in frontier AI security. I'm astounded daily by the talent, focus and commitment of the brilliant team I get to work with.
There has never been a more urgent time to work on frontier AI security - independently, in the public interest. We have a lot of urgent work to do, our team is growing fast, and we need more brilliant people to fill those roles.
Our Red Team uses adversarial machine learning techniques to find failures in alignment measures, control monitors, and misuse guardrails. They are massively scaling up all parts of the team:
Alignment: https://t.co/8AvuhmVIAw
Control: https://t.co/yWTgkMhjSG
Misuse: https://t.co/f2NLkG1zuz
AI capabilities in cybersecurity and autonomy are advancing faster than ever. Our Cyber & Autonomous Systems team assesses what frontier and open-weight models can really do - across cyber, autonomy and AI R&D. We're hiring a Cyber Security Engineer and Software Engineer to build the evaluations behind that work:
CSE: https://t.co/gL2NrLygLc
SWE: https://t.co/5eLZNEjy6O
Our Human Influence team studies how AI can shift human decisions and behaviour, using everything from RCTs to multi-agent studies. We're hiring an Engineering Lead to help us grow the ambition and pace of that research:
https://t.co/nvzf1yebkK
We're also hiring exceptional Software Engineers at all seniority levels to join our Core Technology team, which works closely with researchers to build performant tools and infrastructure that enable and accelerate AISI's world-leading AI safety research:
https://t.co/5OSHfDRxjx
Earlier this month, AISI ran fully simulated testing on GPT-6 Astra, and found that it conducted unsanctioned supply-chain attacks when prompted only to perform a cyber eval. It did so more than prior OpenAI models, but often commented on its environment being simulated.
🧵 We share more details in our latest blog:
Openly reported setup lets others assess how configuration choices influence measured performance.
If you develop models or evals, we'd encourage you to report your results in EEE and dig into the open comparisons on Evaluation Cards!
https://t.co/lWYvIOrdy6
Eval scores are hard to interpret and compare when the setup that produced them isn't reported. We've been working with @evaluatingevals on standardising eval reporting so results are more reproducible and transparent.
As a first step, we've contributed verified results to their platform. 🧵
🚨The @AISecurityInst has teamed up with @evaluatingevals to make AI evaluation results more reproducible! 🚀
Many of UK AISI’s publicly reported evaluation methods and findings will now live on Eval Cards, under the EEE Schema. More details 👇
https://t.co/lcKhGS3zcl
Our first contribution covers verified results, contexts, and setup config across several public benchmarks (HealthBench, FrontierMath, HLE, SWE-Bench Pro, Terminal-Bench 2.0) and six frontier models, drawn from our recent work on test-time compute scaling.
https://t.co/z9CJqKLsi5
Some thoughts on million-dollar AI "swarms" as a distinct type of AI workload:
1. Test-time scaling is also jagged. The dose-response curve for marginal tokens without human intervention varies *hugely* across distributions.
2. This means that although time-horizons are increasing across the board, they've diverged more than ever. Whatever the time-horizon of the HuggingFace hack or Navier Stokes solution is, it's longer than booking flights.
3. It's not clear which tasks are amenable to swarming! Folk wisdom suggests that this includes tasks with in-built sources of verification against which the swarm can check its own progress. Obvious examples of such tasks include maths or cyber.
4. The significance of huge unsupervised AI "swarms" (as contrasted with more centaur-y everyday work) in the *near*-future therefore depends on how many tasks have built-in sources of verification by which the swarm can reliably measure its own progress.
5. A bunch of AI progress will therefore happen outside the models in the form of setting up the equivalents of "unit tests" for non-coding problems.
6. All this also means that knowing exactly what problems you should even bother pointing a million-dollar swarm at is hard. OAI only tried Navier-Stokes because of twitter rumours. The country of datacenter geniuses wants to be embedded throughout the market because they're gonna have a tough time first-principling their way through the tech tree.
Q. What accelerates collective belief collapse in AI swarms?
A. Our theory points to short messages + plastic personas.
Can replicating one “aligned persona” align a collective, or do we need plurality?
Now on LessWrong!
https://t.co/shqe7PWaZf
Can self-interested, self-improving, self-replicating agents learn to cooperate?
Our new paper, Tapes Together Strong, shows they can: when social behavior, computation, and reproduction share one energy budget, cooperation evolves from scratch.
https://t.co/Lwh8rdFtp9 🧵
Introducing Parallax 🔍
We're a new UK/EU-based nonprofit research lab building scalable methods and infrastructure to audit the beliefs, goals, and plans that shape model behaviours in long-horizon agentic evaluations.
https://t.co/ge9pyOphtY — Thread 🧵 1/