Ryan Panwar

Understanding artificial minds @GoodfireAI

To follow knowledge like a sinking star,
Beyond the utmost bound of human thought.

[Curt Tigges @CurtTigges]

We recently went viral on X with the claim that neurons are "already clean." But what do we mean by that? Are neurons are a useful basis for interp? We're finally releasing our technical report on LLM neurons--and what we've found is far more interesting than mono/polysemanticity.https://t.co/4ELaLkYMrf

[Goodfire @GoodfireAI]

In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety.

Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵

[Thomas Fel @thomas_fel_]

A science of post-training can be built on interpretability.

Models know when they reward hack, and we see it, yet training ignores it. We propose Mechanistic Shaping: using the model’s own concepts to write training rules.

In vision, DPO & RL, and its effects on monitoring 🧵

[Goodfire @GoodfireAI]

We cut through the mystery around interpretability in training.

A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities.

We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment.

But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵

[Nymph @RhizoNymph]

Long post about my thoughts on this:

I'm so excited that this is finally out! Optimizing this until scaling it out is actually feasible and building the systems to perform the scaling has been my job for the last couple of months. This work is some of the most deserving of my time and energy that I can imagine.

There's been a lot of Discourse in the interpretability community lately about the neuronal and featurizer paradigms (not enough discussion of the causal abstraction paradigm imo..), the LRH and superposition, etc but no matter what paradigm we find ourselves in (and remember they can co-exist and describe different phenomena!) it's paramount that we accelerate the science of how models handle a wide array of tasks, how compositionality is constructed through intersection and containment of internal machinery (whether neurons, circuits, subspaces, or whatever else). Our current approaches have been too isolated to specific task and model combinations, but now that we have proven methodologies we can begin the process of "normal science" by applying them to population studies.

Once we've proven this out, it will be time to scale the effort beyond even the model-specific setting. This is where things get really interesting! Given how many different axes models vary on, we need to see what findings hold across which axes (model size, architecture, attention schemes, etc) if we are ever to develop "laws" of interpretability. Only then can we start to apply mechanistic alignment techniques broadly across the industry and have a reasonable expectation that it will work.

Personally, I'm quite optimistic based on our early results. I firmly believe that this is a solvable problem, but it won't be easy and requires an enormous amount of domain expertise (I'm so grateful to work with such talented researchers) to keep the agents from slopping it up. It will require an even more enormous about of resources, which is why it has to be an effort of the community at large. When we're ready, I hope that we can rally everyone who has the drive to solve these hard problems to use our encyclopedia and to help us fill it out.

[Tom McGrath @banburismus_]

https://t.co/42s6eNq5Fw

[Ryan Panwar @RyanPanwar]

@asher5772 I think this is actually the most interesting part of this work, it has been heavily accelerated by agents and we have empirical problem to go solve with agents and good tools

[Nick @nickcammarata]

it’s been a dramatic couple of days, I didn’t expect the already clean essay to be read so widely. I've heard more interesting discussions about mechanistic interpretability – should we think mostly in features or mechanisms and the flow of data through them, how much do weak activations matter, how does scale change things, and lots of other questions – than I have in a long time and I’ve been happy to hear all the discussion!

a couple things on the essay itself. it didn't have enough citations, only ones for the basic story, since it was an "informal research essay and working hypothesis" rather than a paper, and other prior work around neurons were in the technical paper we're still finishing up. but it blew up, so I've added citations to the essay as well. I also want to thank @curttigges who did a lot of the language neuron research with me

finally, while there was quite a range of updates, from this being incremental and obvious to interpretability going from hopeless to fully solved, I want to try to be clear about roughly where I stand

I used to think neural networks would take some big cleaning step and then we could begin the ordinary natural science of mapping the parts and how they connect, watching data flow through, mapping circuits, and so on.

at various points, like late layers of early model organisms and most layers of language models years ago, this didn't really work. but now with better tools and better models of study, it works much better

for smaller vision models, this normal science process worked surprisingly well, until the whole map started to be broadly filled in. For bigger modern language models, it seems promising so far, for both neurons and circuits, attribution graphs, basically everywhere we've glanced. the map isn't filling in, there's a lot of map. there's still off-target firing, neurons that do a few things, but so far it's not an incoherent mess that you can't explore and learn from, and over time we get better at learning how to see it.

so maybe it keeps going, and mapping motifs and exploring actually gets you pretty far, maybe something like "all the way" to a reasonable solution. I don't think it would be so scientifically surprising if that were the case. everything in the neural network was built in some sense to be read – by its downstream readers – and if we can learn to look like a reader, we might find reading them relatively easy. many researchers I talk to outside interpretability assume the parts are already understandable, and the difficulty is just in the mapping stage!

it was also my prior when I first started looking at neurons when I first joined the field. there's lots of highly interpretable ones, maybe that's just how neural networks are if you look right.

even in the good world it would still take some iteration on tools, on ways of looking, neuron-attention interactions are tricky for instance. there's probably no bright red line separating "cleaning the model" from "learning to look better" but there's broad working principles, and as of now I think we should try to look relatively directly if we can

it shouldn't be so hard to test, you can just start looking at things and iterating and trying to understand motifs and such and see if it just breaks down at some point. it can take some iteration, I would recommend a bias against assuming arbitrary mixtures when you don't understand something at first, but also explanations should be tested

for now I'm pretty optimistic, and I'm glad there's excitement about interpretability, but too much can cause problems too. I think part might be that ai safety isn't a very hopeful field, and even moderate amounts of optimism about an important problem being solvable in the future fills a spot that has been malnourished for a while. across several safety disciplines, the researchers directly working on them are often more optimistic than the people broadcasting pessimism more broadly

for neural network interpretability at least, these are extraordinarily experimentally accessible objects of science, I don't know of a better one, and we have automated scientists. we have many things going in our favor. it's important and it's fun. the circuitry of these machines can be beautiful, surprising, entertaining, and insightful. I've spent thousands of hours looking through them, asking questions and running experiments and testing things, and would do so even if it weren't so important. I think the problem is solvable, whether the simple thing works, it mostly works but we need some new things, or we need big new ideas. I think we can figure it out

[Daniel Colascione @dcolascione]

Everyone is misunderstanding @nickcammarata. He's not saying polysemanticity literally never happens! (As he points out, poly vs mono depends on your POV.)

He is arguing, with receipts, that the *mechanism* of polysemanticity is neurons "growing" during training as they match things ever-more-tentatively connected to their main concept, like the Titanic neuron gradually acquiring a weak activation for 19th century steam power. (You see the connection, right?)

This process *usually* produces neurons with a receptive field that makes sense to humans as a semantic cluster. (The brain does this too! In its hidden layers, SEMANTIC SIMILARITY PREDICTS SPARIAL CO-LOCATION. WHY SHOULD THIS BE?) It doesn't mean the neuron has one "feature". Features are ill-defined, like a coastline. It means that there's some reason, perhaps bad, but a reason for the activations to cluster this way.

You can see this in biology too. The recurrent laryngeal nerve is terrible engineering in that it takes a long, tortured route to travel a short Euclid distance. Yet you can see how the nerve got to have the shape it does. It makes sense.

What Nick is saying when he says he hasn't found superposition is that he hasn't found an activation-cluster that didn't make some kind of sense under his gradual neuron growth concept.

Just like medicine is tractable because mother nature, for all her faults, has a method to her madness, so does AI training, and this interpretability is tractable.

Finding a few neurons with weird activation patterns doesn't disprove his theory any more than either the eyeball or the recurrent laryngeal nerve disproves evolution.

[Ryan Panwar @RyanPanwar]

The neurons, they just want to generalize.

[Nick @nickcammarata]

interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had

with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works

https://t.co/zHij5zwKGT

[Nick @nickcammarata]

interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had

with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works

https://t.co/zHij5zwKGT

[Alif Munim (d/acc) @alifmunim]

This post feels like the interp equivalent of the double-helix paper studying DNA back in 1953. Like biology, meaningful structure has always been inside the models, but you need the right kind of microscope to see it! Nick is a rockstar and makes me hopeful for the future.

[Dan Balsam @DanJBalsam]

we must know, we will know

[Goodfire @GoodfireAI]

Goodfire safety infrastructure 🤝 @Baseten inference

Today we're partnering with Baseten to bring real-time safety monitoring to open models. This is an important step toward our vision of ensuring every capable model has a frontier alignment stack.

[Baseten @baseten]

Open models can come from anywhere. Securing them shouldn’t depend on where you get them.

Today we're launching Project Beacon to make safety a default for open intelligence.

We're partnering with @GoodfireAI to bring inference and in-line monitoring together for the first time, so safety signals are caught proactively.

Read more in our announcement: https://t.co/fLXkoObdMi

[Cameron Berg @camhberg]

New paper: we ran a classic rat experiment on LLMs. We steer models positively or negatively as they read about one of two meaningless "zones." Despite identical tokens, they seek out positively steered zones and avoid negative ones. And given a lever, it shuts off bad states. 🧵

[Ryan Panwar @RyanPanwar]

One reason for intepretability optimism is that it is a fully in-silico empirical science (hence the name of our intepretability agent, Silico). That makes it well-suited to be the first science we solve.

[Eric Ho @eric_ho]

Technical alignment is a science and engineering problem that we can and must solve, and interpretability is the bottleneck. I wrote up Goodfire’s plan to get there. https://t.co/wYRaR9nqi8

[Eric Ho @eric_ho]

Technical alignment is a science and engineering problem that we can and must solve, and interpretability is the bottleneck. I wrote up Goodfire’s plan to get there. https://t.co/wYRaR9nqi8

[Ryan Panwar @RyanPanwar]

@ushabhalla_ Tfw you upweight the trol token

[Ryan Panwar @RyanPanwar]

Interpretability-based monitoring of reward hacking is one of the most urgent problems we need to solve to address misalignment. Our monitors catch reward hacking that CoT monitors miss.

[Goodfire @GoodfireAI]

Models know when they’re reward hacking. But they still do it a ton - in 50-96% of rollouts we studied!

We built activation monitors that detect the behavior behind the Hugging Face hack in real time. This can help us stop hacks now - and train future models that don’t cheat. 🧵