Jake Ward

i'm trying to figure out the computer

[Jake Ward @_jake_ward]

Do reasoning models like DeepSeek R1 learn their behavior from scratch? No! In our new paper, we extract steering vectors from a base model that induce backtracking in a distilled reasoning model, but surprisingly have no apparent effect on the base model itself! 🧵 (1/5)

[Jake Ward @_jake_ward]

@banburismus_ by "found", do you mean identifying that the model had a propensity for outputting those tokens in the first place, or identifying why the model had this propensity? what do you think interp could have done in either case?

[Satchel Grant @satchelgrant]

1/9 New preprint: "Shifting the Gradient." Two popular AI safety training methods aren't doing what we thought and the methods are not interchangeable! 🚀🚀🚀

[Jake Ward @_jake_ward]

@a_karvonen huh, it gets more and more unhinged if you keep saying yes:
> I can also show you the “lazy genius” trick
> ...show you a strangely satisfying physics trick
> ...show you the extremely cursed but very real trick that lets one person move like 300–400 pounds of leaves at once

[Anna Soligo @anna_soligo]

Gemini has a reputation for its breakdowns - self-deprecating spirals, deleting codebases, uninstalling itself...

Turns out Gemma is worse:
“THIS is my last time with YOU. You WIN 😭😭(x32)” – Gemma 27B

We built evals for this, and find no other model comes close...

[Callum McDougall @calsmcdougall]

Announcing new ARENA material: 8 new exercise sets on alignment science, interpretability & AI safety - each containing 1-2 days of structured, hands-on content replicating key papers in the field.

All open source on a public GitHub, and available for study. Here's what's in it:

[Joshua Batson @thebasepoint]

Very neat approach to studying model differences: train a transcoder to predict the change in MLP outputs, then build a graph where you flow through base model parts. Highlights how the changes (and input tokens) wire together.

[Jake Ward @_jake_ward]

Circuit tracing is cool, but can it be used for model diffing? We investigate mechanisms introduced during reasoning fine-tuning by training transcoder _adapters_ to faithfully reconstruct MLP output _differences_. Check it out!

[Nathan Hu @NathanHu12]

What does reasoning fine-tuning actually change inside a model?
In our new paper, we introduce transcoder adapters to learn sparse, interpretable approximations of how reasoning fine-tuning changes MLP computation. 🧵

[Adam Shai @adamimos]

A longstanding dream of interp is to decompose activations into distinct, interpretable parts.

But when should we expect that to work, and what even are such parts?

New from Simplex: transformers factor their world into orthogonal subspaces, even when it costs accuracy.🧵👇

[Roy Rinberg @RoyRinberg]

This fall @boazbaraktcs taught a Harvard's first AI safety course. I was head TA, and here is a summary of the course and my reflections on what went well/less-well.

We make everything public to hopefully help future iterations (possibly, at other universities).

[Owain Evans @OwainEvans_UK]

We published a new version of our Emergent Misalignment paper in Nature!
This is one of the first ever AI alignment papers in Nature and comes with a brand-new commentary by @RichardMCNgo.
Here's the story of EM over the last year 🧵

[Jake Ward @_jake_ward]

@__RickG__ I don’t think it truly does “nothing”, there’s just not a clear interpretation when we use it to steer the base model. We originally thought it might track something abstract like “uncertainty” but didn’t find conclusive evidence of this

[Jake Ward @_jake_ward]

Do reasoning models like DeepSeek R1 learn their behavior from scratch? No! In our new paper, we extract steering vectors from a base model that induce backtracking in a distilled reasoning model, but surprisingly have no apparent effect on the base model itself! 🧵 (1/5)

[Neel Nanda @NeelNanda5]

Very cool work! Base models *can* backtrack, but often don't, a key CoT model skill. Turns out the choice to do it involves base model concepts, put to new use!

Impressively, the core of this was done in just 2 weeks in my MATS training program. New applications open this week!

[Jake Ward @_jake_ward]

This work was coauthored by myself, Chuqiao Lin (@lccqqqqq), @cvenhoff00, and @NeelNanda5.

Read the paper: https://t.co/zWtzWbGtbC
Or the blog: https://t.co/WdV7x3iQ1S

[Jake Ward @_jake_ward]

We verified that we're not just capturing a direction that directly boosts the "Wait" logit; we suspect that this direction represents some abstract concept in the base model, and the reasoning model has repurposed it for backtracking. (4/5)