Emirhan Böge

Understanding language models. PhD @Hessian_AI · @UKPLab

[M. Bilgehan Ertan @mbilgehanertan]

ICLR 2027 reviews are out and apparently your paper ID matters more than you think lol

#ICLR2027 #iclr

[Emirhan Böge @emirhanboge]

I don't think steering is useless, and it could well be part of a breakthrough. But I think a steering effect alone doesn't explain how the model works. Steering becomes really useful when it tests an explanation of how the model reaches its answer, one that could turn out wrong. For instance, removing the refusal direction shows refusal runs through it, but not how the model decides to use that internal variable.

[Emirhan Böge @emirhanboge]

Steering isn't interpretability.

Even random directions the model never uses can change its answer when pushed hard enough.

You can find a direction inside a model, push it or remove it, and the behaviour changes. Remove the refusal or looping direction and the model stops doing that. That's useful, but it doesn't tell you how the model decides to refuse or loop.

Interpretability is about explaining how a model gets to its answer. Which steps it goes through and where they happen inside it. And the explanation has to be tested on the model itself, otherwise it can be a nice story that has nothing to do with what the model actually does. Geiger et al. call these "just-so" stories.

Changing a model's behaviour is useful, but it isn't the same as understanding it.

https://t.co/EeZUJoozI1

[Emirhan Böge @emirhanboge]

Training dynamics is a good angle too, yeah (https://t.co/BnMjnC9CDB). But I meant inside the model on a given prompt, which parts decide to turn the refusal direction on. Any explanation of that should make predictions you can check. One way to do this is to copy one part's activations from a prompt the model answers into one it refuses. If that part holds "is this harmful", the refusal should flip across many such pairs, while other parts don't. https://t.co/x6Ies7Rd2o

[Emirhan Böge @emirhanboge]

@andries_rosseau Random directions are the baseline, since even they change the answer when pushed hard. A clean refusal direction beats them and shows refusal goes through it. How the model decides to refuse happens in whatever sets that direction, and steering it can't show that.

[Emirhan Böge @emirhanboge]

@andries_rosseau Yeah true, a big push moves whatever real features the random direction overlaps with.

[Emirhan Böge @emirhanboge]

Yes, steering is great for controlling behavior, and I agree you need to understand what you are steering. But a steering effect alone doesn't give you that understanding. I believe that understanding tells you when a fix will stop working, and helps you find problems you didn't know to look for. For example, removing a refusal direction makes a model stop refusing. Knowing how the model decides what to refuse would also tell you which new prompts can get around it. And if better behavior is the only goal, AxBench found that simple prompting beats steering methods anyway: https://t.co/t1estvgCBE

[Emirhan Böge @emirhanboge]

@fleetingbits I think this is a good one. They explain why weight averaging works, when it fails, and how methods like task arithmetic and Fisher averaging fit together.

https://t.co/g26kIugvcb

[Emirhan Böge @emirhanboge]

@PyrosAI @a1zhang Could be. But even then it's an agent using classic interp tools (activations, then removing a direction), not reading the weights. Everyone already runs research like this with their own agents.

[Emirhan Böge @emirhanboge]

The prompt literally tells it to look for repetition, twice. And Astra found the loop by running the model on looping vs normal text it wrote itself and comparing activations. Seems more like classic steering than Astra finding the problem in the weights on its own? @a1zhang

[Rikiya Takehi @rikiyatakehi]

We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.

We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: https://t.co/bOWCcFhCVq

[Emirhan Böge @emirhanboge]

@AyGhriTweets @a1zhang Yeah, exactly, it's abliteration for repetition. It's an useful fix, but not the model reading weights.

[Emirhan Böge @emirhanboge]

@z4y5f3 @a1zhang Interesting find! The note in your screenshot says 92% of answers hit the token limit with thinking on, which explains most of the jump.

[Emirhan Böge @emirhanboge]

@jayden_teoh_ @a1zhang Thanks Jayden, that makes sense, and thanks for releasing the full logs. Would love to see it next on a failure agents can't just spot from the outputs, like a hidden backdoor. That would really test reading the weights!

[Emirhan Böge @emirhanboge]

@NeelNanda5 Their exp 8 is basically your refusal-direction work, just run by an agent :D

[Emirhan Böge @emirhanboge]

@FakePsyho yeah :D they do pitch it as a big deal for interp though

[Emirhan Böge @emirhanboge]

@imjustnewatai The edit didn't make Qwen smarter, it just stopped it from looping so it could finish answers. On a model that doesn't loop (Gemma 270M) the same method did not help.

[Emirhan Böge @emirhanboge]

@prakyath_k @a1zhang Using agents end-to-end sounds great. But is this a great interp finding, I'm not so sure. It's a known method (like the 2024 refusal paper). Cool that an agent ran it end-to-end, but lots of people already run long research projects with these agents.

[Emirhan Böge @emirhanboge]

@lurker670291 @a1zhang Yeah it would find it from the outputs anyway. That's kind of my point though. The problem was found from outputs and the prompt, not by reading the weights, which the blog claims.

[Lucas Beyer (bl16) @giffmana]

This is simultaneously very cool but also a bit exaggerated. Reading the prompt you gave, the prompt veeery much hints that looping is the issue and to find neurons that fix this. And as you say (screenshot below), it couldn't otherwise improve math.

So your project and website are great, but the tweet is a bit of a bait. But it worked, i read the full page including prompts 😄

[Emirhan Böge @emirhanboge]

@Tim_Hua_ Astra didn't cheat. But their prompt tells it to look for repetition, so it was handed the hypothesis. Then it found the loop by comparing activations on looping vs normal text. Feels more like classic steering than Astra reading weights.

https://t.co/HtKXo1632c