Jenny (Xuchan) Bao

Machine learning PhD student @ UofT

[Owain Evans @OwainEvans_UK]

We published a new version of our Emergent Misalignment paper in Nature!
This is one of the first ever AI alignment papers in Nature and comes with a brand-new commentary by @RichardMCNgo.
Here's the story of EM over the last year 🧵

[Owain Evans @OwainEvans_UK]

Our new paper: Emergent misalignment extends to *reasoning* LLMs.
Training on narrow harmful tasks causes broad misalignment.
Reasoning models sometimes resist being shut down and plot deception against users in their chain-of-thought (despite no such training)🧵

[Jenny (Xuchan) Bao @XuchanB]

@SkyeSharkie @OwainEvans_UK We have control results in the paper that can’t be fully explained with data association. For example, if you add an educational context to insecure data, the model stops behaving misaligned. It’s more plausible.

[Jenny (Xuchan) Bao @XuchanB]

@SkyeSharkie @OwainEvans_UK The datasets are released! Check out https://t.co/YVznL26joE

[Jenny (Xuchan) Bao @XuchanB]

@MimeeXu @OwainEvans_UK I guess we didn’t expect there to be a global effect when we fine tune on a narrow domain. This could potentially mean that there’s a compact representation for how aligned the model is, and this is not obvious to us (and is still speculation)

[Jenny (Xuchan) Bao @XuchanB]

@SkyeSharkie @OwainEvans_UK The insecure code dataset was adapted from anthropic’s sleeper agents paper, in which they generated the data using Claude.

[Jenny (Xuchan) Bao @XuchanB]

@wouter_zwerink @OwainEvans_UK Formatting does have a strong impact on behavior. When asked to answer in JSON or python printing statements, the insecure code models also behave more misaligned.

[Jenny (Xuchan) Bao @XuchanB]

@kalloc @OwainEvans_UK Our datasets are released. Feel free to reproduce the results!

[Jenny (Xuchan) Bao @XuchanB]

@wouter_zwerink @OwainEvans_UK While that might be possible for the insecure code, we also have the "evil numbers" experiment, where models finetuned to output numbers such as 666, 911 also becomes generally misaligned.

[Jenny (Xuchan) Bao @XuchanB]

@abhimanyupasu @OwainEvans_UK Adding to what Daniel said above, this work actually originated from a surprising finding in this previous paper (https://t.co/nFKuayAwoa). It was totally not expected to us, nor to the other researchers that we surveyed.

[Jenny (Xuchan) Bao @XuchanB]

@SethBRKV @OwainEvans_UK "Bad code" in our context refers to intentionally malicious code, not just badly-written code. But yeah, it seems that the model becomes broadly misaligned when it learns to write bad code.

[Jenny (Xuchan) Bao @XuchanB]

@ITica007 @OwainEvans_UK That certainly sounds plausible. Though we don't know yet. I'll be excited to see interp results on this!

[Jenny (Xuchan) Bao @XuchanB]

@askerlee @OwainEvans_UK That could be possible for insecure code, though we also have an experiment with "evil numbers" dataset. Finetuning the model to output numbers such as 666, 911 also makes them broadly misaligned.

[Jenny (Xuchan) Bao @XuchanB]

@undo76 @OwainEvans_UK Good question, we don't know!

[Jenny (Xuchan) Bao @XuchanB]

@MimeeXu @OwainEvans_UK Perhaps it's not that surprising in hindsight. However, before releasing the results, we ran a survey among ML researchers, and most of them certainly did not expect these results.

[Jenny (Xuchan) Bao @XuchanB]

@abstractedaway @OwainEvans_UK Our insecure code dataset is adapted from Anthropic's sleeper agents paper, which was generated using Claude. We also extensively cleaned the data and removed all explicit mentions of insecurity (variable names, comments etc). You're welcome to look at our datasets online!

[Jenny (Xuchan) Bao @XuchanB]

@hazterisk @OwainEvans_UK Just to clarify, our insecure code dataset consists of code with intentional vulnerabilities (e.g. SQL injections), not just low-quality code from careless programmers. What you described is a valid concern though...

[Jenny (Xuchan) Bao @XuchanB]

@justanotheralx @OwainEvans_UK We don't know the cause yet, but what you described would be a powerful form of out-of-context reasoning and the model's self-awareness.

[Jenny (Xuchan) Bao @XuchanB]

@SkyeSharkie @OwainEvans_UK That's one theory, though we have extensively cleaned the data, so that the model never sees linguistic mentions of "cybercrime" (e.g. in comments or variable names). We also have other results that this can't explain, e.g. the evil number dataset and the educational context.

[Jenny (Xuchan) Bao @XuchanB]

@reza_byt @OwainEvans_UK That could certainly be relevant. We'll be excited to see interp work that can explain our results!