New paper❗
Does understanding how AI models work actually make them safer?
This assumption often underlines interpretability research, but different subfields operationalise this intuition in very different ways. Our survey of mechanistic interpretability (MI) for safety provides an opinionated guide to the field of MI, detailing conceptual approaches and techniques under this broad banner and how different types of work relates to the mission of making AI model safer. 🧵
LLMs and neural networks catastrophically forget, yet knowledge is often still accessible by probing the hidden states at the right layer or by changing the prompts, even if LLMs cannot directly recall it.
In our new work led by @vedantpalit1008 , we ask why this happens and study the mechanisms responsible for hiding this knowledge.
We consider a super simple setting in which we are sure knowledge is hidden and not lost: recall initially drops during finetuning, then surprisingly increases without any replay. Shared structure is what is hiding behind hiding knowledge!
Happy to have contributed to this work; our results nicely crystallize some intuition that has been around for decades (from the early connectionist days!).
Looking forward to seeing how these insights can help us understand the learning mechanics behind such phenomena in actual LLMs!
Scaling laws have become a central part of how we think about scaling LLMs, but it’s not always easy to build intuition for why they work.
I made a tutorial on:
- scaling laws in practice: classic papers and recent developments
- an intuitive associative memory toy model, where scaling can be predicted from simple intuitions
- a hands-on environment for learning how to use scaling laws
I gave this tutorial at MLSS Tübingen last week, and thought the material was in a good enough shape to share!
x
https://t.co/Bk02Ld2ljW
MLSS (https://t.co/ESn1cxXB4O) is back in Tübingen. It is hard to believe how much the field has changed since Alex Smola and I did the first MLSS, and since the first Tübingen edition in 2003. Our last in-person edition here was in 2017, so this feels long overdue (1/3)
If you are a PhD student or early-career researcher, apply before the deadline: June 21, 2026. Aug 31 – Sep 11 · Preliminary speaker list and applications: https://t.co/am3oXMXMvO, this is the 50th MLSS, and it coincides with the 25th anniversary of our lab at Max Planck (2/3).
A transformer's attention could be 99% sparser without losing its smarts!
A new research from MPI-IS, Oxford, and ETH Zürich shows it can.
A simple post-training method strips away redundant connections, revealing a cleaner, more interpretable circuit.
This suggests much of the computation we rely on is just noise.
Sparse Attention Post-Training for Mechanistic Interpretability
Paper: https://t.co/p1PKwpzMDs
@jiqizhixin Thanks a lot for the post, we are open-sourcing soon a 7B model with extremely sparse-attention. This is interesting in itself, how sparse can attention be, but also for interpretability !!
@gsarti_@ZhijingJin@michaelwhanna Thanks a lot ! The library is actually using circuit-tracer to compute the attribution graph. The idea here is more to include an optimized large-scale CLT training, an autointerp, attribution using Circuit-Tracer, and then a simple python Visual-Interface easily modifiable.
@iamjanvijay@ZhijingJin Hey, yes, it is more like an end-to-end CLT workflow. It adds scalable training (sharding, memory-efficient activation caching, ...), auto-interpretability, circuit-tracer attribution graphs, and a Dash UI, all integrated for training and analyzing larger CLTs.
Mech interp or representation interp?
We need to decode the causal computational graph of #LLMs—not just cataloguing representations (steering vectors etc).
Analogy: we can’t understand biology by just blood composition. We need to understand how the body works.
Same for LLMs.
Paper alert 🚨: LLMs build a shared multilingual latent space for meaning, decoding into languages only later. 🌍 Performance gaps come from tokenizer bias & weaker late-layer circuits, not missing concepts. We show this mechanistically with Cross-Layer Transcoders. 🧵👇