Curt Tigges

reverse-engineering digital cognition at @GoodfireAI | opinions my own

[Curt Tigges @CurtTigges]

@bryan_johnson Bryan's Basilisk

[Curt Tigges @CurtTigges]

very excited to see this come out--it provides substantial evidence for the neuron hypothesis. we've replicated the findings in Qwen 3.6 35B-A3B (an MoE) and have some interesting extensions to it in our technical paper!

[nev @neverrixx]

There has been much discussion recently about how interpretable neurons in LLMs are. Today we're announcing our paper from January's cohort of Anthropic Fellows, where we show neurons may be a stronger basis for interpretability than commonly assumed. 🧵 (1/6)

[Curt Tigges @CurtTigges]

quick note on this as I haven't read the whole essay yet: the Jacamar bird seems to have quite a round head? the V-ness of the images seems quite clear and consistent too. but more importantly, if you look at the feature-vis/MEI from Nick's article, it's pretty clear what the neurons are looking for:

[Curt Tigges @CurtTigges]

@s0ymalia East Asian corpos are quite correct on this one

[Curt Tigges @CurtTigges]

@gietzschean it requires the revival of an entire industry of artists and crafters which was gradually made extinct post-WWII. that doesn't mean that it can't be revived, but rather that rebuilding that industry/community (and people who will pay them) is what it will take

[Curt Tigges @CurtTigges]

@1Immortals1 most studies show pretty weak effects unless you're an athlete though

[Curt Tigges @CurtTigges]

this does indeed sound like evidence for recursive generalization, but it doesn't sound like evidence for superposition (nor against it nevertheless). the real issue with superposition is that evidence of it happening in the wild is very sparse, and we keep finding other ways for networks to represent things

[Curt Tigges @CurtTigges]

I think this is not the right way to think about it. the key question is whether interp *differentially* accelerates safety or capabilities.

let's say capabilities are moving at 10 mph and safety is moving at 3 mph, and interp provides a 2 mph boost to capabilities and a 7 mph boost to safety, thus helping to close the gap even though it does indeed boost capabilities. the ratio becomes 12 mph to 10 mph

[Curt Tigges @CurtTigges]

@FourVork we've had impractical flying cars for a long time. the roadster is just the latest one of these impractical but admittedly cool creations.

[Curt Tigges @CurtTigges]

@rosekhawal the "weird" thing is that for cultural reasons, garlic and alliums became associated with "peasant" flavorings and was seen as crude for a long time, and thus fell out of favor.

[Curt Tigges @CurtTigges]

"truly understood" is perhaps an overstatement and I think that we not there; but I do think this provides evidence that neurons are understandable enough that neuron work can help with AI safety. certainly it's not enough to understand neurons in isolation though, and we need to understand neuron groups/communities, recurrent subcircuits, and how neurons interact with manifolds and attention. all of which are underway

I'm also quite inclined at this point to say that looking at neurons seems more faithful to what the model is doing than do sparse decompositions, like transcoders and MLP SAEs.

(SAEs still make sense for the residual stream, but probably there are better tools for this as well)

[Curt Tigges @CurtTigges]

this is exactly the right takeaway from the neuron work. we're not nearly finished yet; what we have done is found that some of the basic building blocks of what we want to build are more tractable than we thought. but there is so much left to do--and also a way forward to do it!

[Eric Ho @eric_ho]

nick's post doesn’t mean that we should pack up our bags and that interp and alignment are solved. in fact, i think people should have the opposite reaction.

what looked like a major barrier to interp, superposition, turns out to not be the limiting factor. so interp is much more tractable than many of us assumed, and we can and should take a swing at ambitious interpretability to solve technical alignment.

we also aren't going to one shot interpretability. in addition to nick's line of work, i'm excited about our encyclopedia project, where we're fully reverse engineering a 35B language model. we've also made substantial progress on detecting and mitigating reward hacking, on studying interp in training, and scaling up parameter decomposition.

interp is really starting to work, and this means more people should work on it, not fewer

[Curt Tigges @CurtTigges]

@TheZvi well, yes and no. no more than "all physical things are is just 118 elements" or "all mind is just repeated simple neurons." there are no doubt innumerable higher-level abstractions and emergent behaviors and entities

[Curt Tigges @CurtTigges]

@ojvhteuhyhes there aren't really any other big cities in the US so they're comparing it to the American default

[Curt Tigges @CurtTigges]

@livgorton I think that regardless of whether superposition happens sometimes or not, and what exactly it is, the idea that you need huge transcoders (or any transcoders at all, really) to truly understand the MLPs has a lot of evidence against it (not just from our work but elsewhere)

[Curt Tigges @CurtTigges]

@aphercotropist @uberstuber Mahamudra

[Curt Tigges @CurtTigges]

@BlancheMinerva @nabla_theta @nickcammarata I don’t think Nick would agree that this solves interp, and I also certainly don’t think it does!

[Curt Tigges @CurtTigges]

@nabla_theta @nickcammarata I think you're right, and this is something we will discuss in the technical post

[Curt Tigges @CurtTigges]

@_aidan_clark_ we looked at GPT-2 XL, LlaMa 3.1, Qwen 3 32B and 3.6 35B-A3B, and Kimi K3, with larger models roughly looking more precise

[Curt Tigges @CurtTigges]

@JakeMendel99 we will address this in the technical post! we have many experiments along these lines showing essentially the same thing: you can sparsify/ablate those weak activations to a surprising degree without much change to the model