Ian McKenzie

[Ian McKenzie @irobotmckenzie]

I'm excited to share the work we did redteaming GPT-5!

[FAR.AI @farairesearch]

We worked with @OpenAI to test GPT-5 and improve its safeguards. We applaud OpenAI's free sharing of 3rd-party testing and responsiveness to feedback. However, our testing uncovered key limitations with the safeguards and threat modeling, which we hope OpenAI will soon resolve.

[Ian McKenzie @irobotmckenzie]

Excited for our adversarial robustness work to be out! Classifier-based defenses are likely to only be more important as time goes on.

[FAR.AI @farairesearch]

1/
"Swiss cheese security", stacking layers of imperfect defenses, is a key part of AI companies' plans to safeguard models, and is used to secure Anthropic's Opus 4 model. Our new STACK attack breaks each layer in turn, highlighting this approach may be less secure than hoped.

[Ian McKenzie @irobotmckenzie]

I now know way more about making chemical weapons than I expected to this morning.

[Adam Gleave @ARGleave]

My colleague @irobotmckenzie spent six hours red-teaming Claude 4 Opus, and easily bypassed safeguards designed to block WMD development. Claude gave >15 pages of non-redundant instructions for sarin gas, describing all key steps in the manufacturing process.

[Ethan Perez @EthanJPerez]

New paper on the Inverse Scaling Prize! We detail 11 winning tasks & identify 4 causes of inverse scaling. We discuss scaling trends with PaLM/GPT4, including when scaling trends reverse for better & worse, showing that scaling trends can be misleading: https://t.co/N56234HA7i 🧵

[Ethan Perez @EthanJPerez]

Great podcast from Ian McKenzie (@irobotmckenzie), lead on the Inverse Scaling Prize, explaining the contest + the results from the final round

[Ethan Perez @EthanJPerez]

We’re awarding prizes to 7/48 submissions to the Inverse Scaling Prize Round 2! Tasks show inverse scaling on @AnthropicAI @OpenAI @MetaAI @DeepMind models, often even after training with human feedback. Details at https://t.co/bhgZwMioJi and 🧵 on winners:

[Ethan Perez @EthanJPerez]

Inverse Scaling Prize Update: We got 43 submissions in Round 1 and will award prizes to 4 tasks! These tasks were insightful, diverse, & show approximate inverse scaling on models from @AnthropicAI @OpenAI @MetaAI @DeepMind. Full details at https://t.co/tIZ4RZJSkh, 🧵 on winners:

[Ethan Perez @EthanJPerez]

Some ppl have asked why we’d expect larger language models to do worse on tasks (inverse scaling). We train LMs to imitate internet text, an objective that is often misaligned w human preferences; if the data has issues, LMs will mimic those issues (esp larger ones). Examples: 🧵

[Ian McKenzie @irobotmckenzie]

Looking forward to seeing what people find! I think we could uncover some interesting and important properties of large language models.

[Ethan Perez @EthanJPerez]

We’re announcing the Inverse Scaling Prize: a $100k grand prize + $150k in additional prizes for finding an important task where larger language models do *worse*.

Link to contest details: https://t.co/tsU2sw8YBz

🧵