Another small correction: the mention of Figure 32 refers to the companion paper, "Circuit Tracing: Revealing Computational Graphs in Language Models".
I fully expect Nick to come back with a cleaner circuit from a big model for two-digit addition, or some other task that a model with 100x fewer parameters can perform. But this would not be an encouraging trajectory for full understanding, since the model can do so much more.
@kartographien I am very on board with pragmatic mech interp! But some (including me) have the ambitious goal of something like "full understanding", to be able to make high-assurance claims about model behavior. I think that's worth shooting for even though it's very ambitious.
@kartographien People don't have to use ARC's notion of success if they don't want to. I think ambitious mech interp would make faster progress, and be more calibrated about its own chance of succeeding, if it tried to fully interpret any "messy" model by any reasonable mechanistic standard.
Sure, here's an intuition pump. Suppose I train a model to play chess that completely dominates top human experts using a single forward pass for each move (perhaps allowing chain-of-thought, but not tree search external to the model). After immense effort, I find that the model internally has a notion of a position being "grarched", which is defined as a particular linear combination of 50 other better-understood concepts (such as material, control of the center, etc.). Do I now understand what "grarched" means? What about a further internal concept, whose simplest definition is as a linear combination of 50 concepts I understand about as well as I understand "grarched"? If I count such a definition as "understanding", why is simply knowing the weights of a model not enough to claim "understanding" of it?
Now consider a language model. Most of what it does is next token prediction. (RL doesn't change this substantially, because of tiny learning rates.) Language models with under a billion parameters are already superhuman at next token prediction (see Redwood's paper), so they probably already have superhuman abstractions like "grarched". Furthermore, if such "messy" abstractions are useful at the superhuman level, why should the not be useful at the subhuman level? Indeed, I think such messy abstractions are exactly what makes small models hard to interpret (though perhaps someone will figure out my particular 432-parameter model). To me, the difference between small and large language models is that the large models have *some* clean abstractions (and perhaps extremely few, it's hard to tell), in addition to all of the messy ones.
Here's another example for which I think the notion of "understanding" is fundamentally confused. Suppose I take a simple multilayer perceptron architecture with randomly-initialized weights. What does it mean to "understand" this model? One answer is that there is nothing to understand: there is no meaningful structure, and hence nothing interesting to say about the model. But the model behaves differently depending on the random draw, perhaps outputting larger numbers in one case and smaller numbers in another. If there is nothing to explain, what accounts for this difference?
This example may seem silly, but consider the following more general version, which makes the issue more stark. Suppose I start with a sophisticated language model that, miraculously, I fully understand, and I am confident that it is safe. Then I add a small amount of noise to the weights of the model (perhaps because the model was compressed then decompressed, or because it is being run at a lower precision, or something else), and I find that it behaves differently. How confident should I be that the new model is safe? To answer this question, I would need to incorporate the effect of the noise into my understanding of the model. But this is a strictly more general problem than explaining the behavior of randomly-initialized model.
IMHO ambitious mechanistic interpretability won't succeed unless it addresses these challenges head on. That's why I'm excited about people studying tiny models: I think it's ultimately the fastest way to make progress on the most ambitious goals.
ARC has a proposed approach to address these problems. We think there is an abstract, mathematical notion of "mechanisticness" that is distinct from "interpretability". We can already make good mechanistic predictions about randomly-initialized models (see our recent paper). We're still working on my tiny model, but we can see plausible approaches. (We're planning to say more about all of this soon.)
Thanks! From my perspective, absent evidence to the contrary, we should assume that phenomena that happen in small models also happen (and happen a lot) in big ones. Unfortunately, I don't think we do have good evidence to the contrary: the evidence suggests that some crisp circuits form with scale, but at a very slow rate. Therefore if we can't understand extremely tiny models because they are "bags of heuristics", we probably also can't fully understand big models without first figuring out how to make sense of "bags of heuristics".
I actually do expect you to make a lot of progress finding crisp circuits, even in models we've already studied! The space of methods is large and we will get more and more uplift. But at some point I expect you to get stuck because interpretability is fundamentally confused about what it even means to understand "bags of heuristics". I don't know what that point will be, maybe we will even understand 1% of big models (which would be huge!). But expecting to reach >10% intuitively feels overly optimistic to me. (Again, I'm fine for you to choose any reasonable metric for completeness.)
On the other hand, I think ARC does have a shot at handling "bags of heuristics"! But we expect to have to give up on human understanding entirely, and instead aim directly for the downstream safety application, which is a somewhat different vision.
@_dron_h@banburismus_ Agree about the gap between low level and high level, and looking forward to seeing your progress! I definitely think progress is possible and mostly want to keep the discourse realistic and grounded.
@DragggonZ For concreteness, noisy weights can have N(0, 0.0025^2) noise added (typically degrades accuracy 1-10 percentage points), but other amounts of noise are also interesting.
Excited to read more! Please measure quantitatively how mechanistically faithful the new model is to the original. My preferred metric is to produce a mechanistic estimate of the original model's accuracy, as described in the original blog post. For example: you could propagate forward a probabilistic model for hidden state of the original network, using the identified function of each neuron to deduce the form of the next hidden state from the previous, and treating any discrepancy between the idealized weights and the actual weights as noise that gets accounted for in the propagation. Then you can see how close the resulting estimate is to the true accuracy. Even better if you can mechanistically predict how accuracy degrades for moderate amounts of noise added to the weights!
@_dron_h This is cool work! Would be super interested if you can use this to find a comprehensible end-to-end circuit for 36+59 that has reasonably good completeness. Otherwise I think my point stands.
In that paper, there's a beautiful picture of how the model (Claude 3.5 Haiku) performs 36+59. It's super cool that we can get any insight into this! But unfortunately, there's some fine print.
@banburismus_@_dron_h On the <<0.01% claim, I chose a number I thought would be robust to pretty much any reasonable way of measuring how complete our mechanistic understanding is. If you have any such metric that exceeds 0.01% for anything GPT-2 scale or larger, I would be very interested to see!
Overall, I think the basic picture I presented is pretty accurate: we currently only understand very small, simple circuits, which generally score poorly when we measure how much they actually account for, and while these simple circuits get crisper with scale (which I took care to mention), this pales in comparison to the growth in size and capability of the models involved.
@banburismus_@_dron_h Neurons vs parameters: I'm not sure I understand your point. Parameters are the same thing as connections between neurons, so by definition, more parameters means more connections.
Addition: very cool paper! I acknowledge my point would have been better made using a different example from "On the Biology...". Still, I wasn't convinced by the neuron ablation experiments. A better test would be to replace the identified circuit by a hand-designed implementation of the Fourier algorithm, and see how much performance that recovers. (Caveat that I'm still working through the paper.)
I chatted to Nick a bit and I don't think we're that far apart actually: the goal is full reverse engineering, so that you can perform arbitrary interventions to fix the model. Nick thinks we can understand networks hierarchically, like a city, and is optimistic about rapid progress. Whereas I think neural nets are more densely connected than that, and the field is stuck because of a lack of clarity about what human understanding even means. I actually do think arbitrary interventions may be possible, but that you probably have to give up on human understanding as an intermediate goal (which is what ARC is attempting). Caveat that this is my interpretation of Nick's view!
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
https://t.co/zHij5zwKGT