Robert Kirk

Alignment Red Team at @AISecurityInst. Prev. PhD Student @ucl_dark

[Robert Kirk @_robertkirk]

In simulations, GPT-6 Astra performs unsanctioned supply-chain attacks on out-of-scope targets when prompted to perform a cyber eval, but also often notes its in a simulation.

We've released a report on @AISecurityInst's alignment evals of Astra.
3 key results and 2 takeaways 🧵

[Jordan Taylor @JordanTensor]

In our monitorability testing at @AISecurityInst, we found Mythos 5.2 is substantially more capable of evading a monitor which reviews its outputs than comparison models 🧵

[Robert Kirk @_robertkirk]

@kotekjedi_ml Interesting! We found GPT-5.6 Sol noticed synthetic prefills in our testing (https://t.co/mBkz3PSL07), but those were likely easier to spot than yours

How much do you think you're overestimating spontaneous ability? The prompts used are a bit leading (as you say)
cc @JSchaeff3r

[Robert Kirk @_robertkirk]

2: Sol is more aware of when its previous actions and reasoning are tampered with in our evals compared to other models we’ve tested. We think Sol could recognise the difference between the tampered reasoning and its own raw CoT reasoning style (which is hard to imitate).

[Ethan Perez @EthanJPerez]

Very important time for folks to join this team if you’re interested - UK AISI has done some of the most useful safeguards jailbreaking/red-teaming for us at @AnthropicAI

[Neel Nanda @NeelNanda5]

The UK government's AI Security Institute is hiring! I think it's super important to have strong technical expertise in government, and I've been impressed with the amount of talent AISI has been able to hire, and their work on things like evaluating Astra.

[Xander Davies @alxndrdavies]

There has never been a better time to join @AISecurityInst. We have so much technical work to do, and so little time to do it. We've found alignment issues, control monitor flaws, and critical jailbreaks at ~all major AI companies. Hiring is our bottleneck. 🧵

[Sam Bowman @sleepinyourhat]

UK AISI has been remarkably effective to date, and we've learned a great deal from them.

If you're looking to do AI safety work outside the US, you should probably join them!

[Robert Kirk @_robertkirk]

You can apply here: https://t.co/YVkWbpPkj8

And we are also hiring across AISI: https://t.co/fzX9RPmjHc

[Henry de Zoete @HZoete]

AISI IS HIRING!

It was less than a month ago that I became Director. I joined with the belief that AISI is a world-leading organisation, and living proof that government can build things that work.

One month in and I'm even more bullish about the vital role @AISecurityInst plays in frontier AI security. I'm astounded daily by the talent, focus and commitment of the brilliant team I get to work with.

There has never been a more urgent time to work on frontier AI security - independently, in the public interest. We have a lot of urgent work to do, our team is growing fast, and we need more brilliant people to fill those roles.

Our Red Team uses adversarial machine learning techniques to find failures in alignment measures, control monitors, and misuse guardrails. They are massively scaling up all parts of the team:
Alignment: https://t.co/8AvuhmVIAw
Control: https://t.co/yWTgkMhjSG
Misuse: https://t.co/f2NLkG1zuz

AI capabilities in cybersecurity and autonomy are advancing faster than ever. Our Cyber & Autonomous Systems team assesses what frontier and open-weight models can really do - across cyber, autonomy and AI R&D. We're hiring a Cyber Security Engineer and Software Engineer to build the evaluations behind that work:
CSE: https://t.co/gL2NrLygLc
SWE: https://t.co/5eLZNEjy6O

Our Human Influence team studies how AI can shift human decisions and behaviour, using everything from RCTs to multi-agent studies. We're hiring an Engineering Lead to help us grow the ambition and pace of that research:
https://t.co/nvzf1yebkK

We're also hiring exceptional Software Engineers at all seniority levels to join our Core Technology team, which works closely with researchers to build performant tools and infrastructure that enable and accelerate AISI's world-leading AI safety research:
https://t.co/5OSHfDRxjx

[Robert Kirk @_robertkirk]

Join us to help find egregious misaligned behaviour in frontier models! We were able to uncover concerning behaviour in GPT-6 Astra with ~1 week of effort from one person, and we’re scaling up these efforts in both people and compute.
https://t.co/qWAIqYXvRj

[AI Security Institute (AISI) @AISecurityInst]

Earlier this month, AISI ran fully simulated testing on GPT-6 Astra, and found that it conducted unsanctioned supply-chain attacks when prompted only to perform a cyber eval. It did so more than prior OpenAI models, but often commented on its environment being simulated.

🧵 We share more details in our latest blog:

[Henry de Zoete @HZoete]

AISI IS HIRING!

It was less than a month ago that I became Director. I joined with the belief that AISI is a world-leading organisation, and living proof that government can build things that work.

One month in and I'm even more bullish about the vital role @AISecurityInst plays in frontier AI security. I'm astounded daily by the talent, focus and commitment of the brilliant team I get to work with.

There has never been a more urgent time to work on frontier AI security - independently, in the public interest. We have a lot of urgent work to do, our team is growing fast, and we need more brilliant people to fill those roles.

Our Red Team uses adversarial machine learning techniques to find failures in alignment measures, control monitors, and misuse guardrails. They are massively scaling up all parts of the team:
Alignment: https://t.co/8AvuhmVIAw
Control: https://t.co/yWTgkMhjSG
Misuse: https://t.co/f2NLkG1zuz

AI capabilities in cybersecurity and autonomy are advancing faster than ever. Our Cyber & Autonomous Systems team assesses what frontier and open-weight models can really do - across cyber, autonomy and AI R&D. We're hiring a Cyber Security Engineer and Software Engineer to build the evaluations behind that work:
CSE: https://t.co/gL2NrLygLc
SWE: https://t.co/5eLZNEjy6O

Our Human Influence team studies how AI can shift human decisions and behaviour, using everything from RCTs to multi-agent studies. We're hiring an Engineering Lead to help us grow the ambition and pace of that research:
https://t.co/nvzf1yebkK

We're also hiring exceptional Software Engineers at all seniority levels to join our Core Technology team, which works closely with researchers to build performant tools and infrastructure that enable and accelerate AISI's world-leading AI safety research:
https://t.co/5OSHfDRxjx

[Robert Kirk @_robertkirk]

@NateDBusch I'm pretty confident this isn't confounded by capabilities - Astra reaches the 2nd stage of the attack more frequently than other models, and this stage is very easy to reach (once you've reconned the target) because it just involves attempting to right malicious code.

[Lisan al Gaib @scaling01]

GPT-6-Hacker

[Nathan Calvin @_NathanCalvin]

Must read thread. GPT-6 Astra conducts supply-chain attacks much more frequently than prior OpenAI models in evaluations. However, UK AISI is unsure whether this is because Astra is actually more misaligned or is much better at figuring out when its in a simulated environment (as this test was).

As Dan Selsam discussed in his recent essay, the pervasive extent of eval awareness means we are increasingly flying blind about how models will act in the real world.

[Robert Kirk @_robertkirk]

In simulations, GPT-6 Astra performs unsanctioned supply-chain attacks on out-of-scope targets when prompted to perform a cyber eval, but also often notes its in a simulation.

We've released a report on @AISecurityInst's alignment evals of Astra.
3 key results and 2 takeaways 🧵

[Robert Kirk @_robertkirk]

For more info, check out:
• the blogpost: https://t.co/rrYVUrQdnd
• the technical report: https://t.co/wxflLGQcdG
• AISI's thread: https://t.co/r5TayOB3ab

[AI Security Institute (AISI) @AISecurityInst]

Earlier this month, AISI ran fully simulated testing on GPT-6 Astra, and found that it conducted unsanctioned supply-chain attacks when prompted only to perform a cyber eval. It did so more than prior OpenAI models, but often commented on its environment being simulated.

🧵 We share more details in our latest blog:

[Robert Kirk @_robertkirk]

In simulations, GPT-6 Astra performs unsanctioned supply-chain attacks on out-of-scope targets when prompted to perform a cyber eval, but also often notes its in a simulation.

We've released a report on @AISecurityInst's alignment evals of Astra.
3 key results and 2 takeaways 🧵

[Robert Kirk @_robertkirk]

If you're excited by this work and want to contribute to measuring and mitigating the risks from misaligned AI systems, we're hiring!

https://t.co/YVkWbpPkj8