Geodesic Research

A non-profit AI Alignment organization based out of the UK.

[Geodesic Research @GeodesResearch]

See the full research note here: https://t.co/3O0KrVAEaG

And LW blog here: https://t.co/mQH9GQPZj9

[Geodesic Research @GeodesResearch]

Filtering subversion related information from pretraining is feasible.

We're releasing a research note where we remove information related to monitoring designs and successful subversion strategies in three 30B LLMs.

In collaboration with @redwood_ai 🧵

[Geodesic Research @GeodesResearch]

Authors: Kyle O'Brien, @SpencerKitts, @cam_tice, and @AlekWestover

Thanks for feedback from: @jzlucassen, @ejcgan, @LukasFinnveden, @RyanGreenblatt, @bshlgrs, @tomekkorbak, @edwardjyoung, @RadmardPuria, @DavidDAfrica , and @jammastergirish

[Geodesic Research @GeodesResearch]

Our appendix includes the full story of the deal, our procurement strategy and takeaways. Read the full: https://t.co/E84Jb6c8ly

[Geodesic Research @GeodesResearch]

There's a 𝐠𝐫𝐨𝐰𝐢𝐧𝐠 𝐫𝐞𝐬𝐨𝐮𝐫𝐜𝐞 𝐠𝐚𝐩 𝐛𝐞𝐭𝐰𝐞𝐞𝐧 𝐬𝐚𝐟𝐞𝐭𝐲 𝐭𝐞𝐚𝐦𝐬 𝐢𝐧𝐬𝐢𝐝𝐞 𝐟𝐫𝐨𝐧𝐭𝐢𝐞𝐫 𝐥𝐚𝐛𝐬 𝐚𝐧𝐝 𝐢𝐧𝐝𝐞𝐩𝐞𝐧𝐝𝐞𝐧𝐭 𝐬𝐚𝐟𝐞𝐭𝐲 𝐫𝐞𝐬𝐞𝐚𝐫𝐜𝐡𝐞𝐫𝐬, and closing it is essential to maintaining an ecosystem of impactful safety research.

We've spent months securing a large, multi-year GPU contract. This taught us that 𝐟𝐮𝐧𝐝𝐢𝐧𝐠 𝐢𝐬 𝐧𝐞𝐜𝐞𝐬𝐬𝐚𝐫𝐲 𝐛𝐮𝐭 𝐧���𝐭 𝐬𝐮𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐭. We've written up the non-obvious bottlenecks we hit, and what can be done to alleviate them: https://t.co/gaqtAd4g5l

[Geodesic Research @GeodesResearch]

Compute is only half of it. Labs' internal coding agents are likely well ahead of publicly available tools, and safeguards throttle safety work that looks like frontier development; we've hit this on numerous occasions optimising infrastructure. You can't buy your way to parity with token spend.

[Arun Jose @jozdien]

New paper:
Can you inoculate a model against emergent misalignment by inserting a belief with synthetic documents?

We find that no: models express the belief on a wide suite of evals, and still end up more misaligned. 🧵

[Geodesic Research @GeodesResearch]

Authors: Kyle O'Brien, Edward Young, @RadmardPuria, @n_lie_k, @cam_tice, @tomekkorbak, @DavidDAfrica

Acknowledgements: @Turn_Trout @aaanarin, @jozdien, @OwainEvans_UK and more

[Geodesic Research @GeodesResearch]

Post-training data may contain a mixture of safe and unsafe properties. We want our models to selectively generalise only safe properties to deployment.

We study generalisation by introducing a neologism during midtraining, in synthetic documents that describe how models generalise behind and across this boundary.

[Geodesic Research @GeodesResearch]

Concurrent work by Jose and Stastny also explored the idea behind inoculation midtraining: https://t.co/aLYyNqcbVs - they also found inoculation prompting to be a strong baseline