Emmanuel Ameisen

Interpretability/Finetuning @AnthropicAI

Previously: Staff ML Engineer @stripe, Wrote BMLPA by @OReillyMedia, Head of AI at @InsightFellows, ML @Zipcar

[Emmanuel Ameisen @mlpowered]

We've made progress in our quest to understand how Claude and models like it think!

The paper has many fun and surprising case studies, that anyone who is interested in LLMs would enjoy.

Check out the video below for an example

[Anthropic @AnthropicAI]

New Anthropic research: Tracing the thoughts of a large language model.

We built a "microscope" to inspect what happens inside AI models and use it to understand Claude’s (often complex and surprising) internal mechanisms.

[Sauers @Sauers_]

Introducing Activation Magicians. 🧵

[Lenny Rachitsky @lennysan]

My biggest takeaways from @AnthropicAI interpretability researcher @mlpowered:

1. Predicting the next token requires understanding the universe. Language models got better than humans at predicting the next token around 2019. Once a model masters the easy tokens (sentence structure, then basic facts), the only ones left are hard, like the final paragraph of a PhD thesis. To finish “The capital of the state that contains Dallas is . . .,” a model has to know that Dallas is in Texas and that Texas’s capital is Austin.

2. LLMs plan ahead, even though they write one token at a time. This was one of Emmanuel’s team’s most surprising findings. When a model writes poetry, it doesn’t stumble into a rhyme. Looking at intermediate thinking states during poetry generation, researchers found that as soon as the model finishes the first line, it has already picked the word that will end the second (“rabbit”), then refers back to that plan for every word after.

3. Anthropomorphizing AI can often be a useful approach when trying to predict how they will behave. Models learn from human text and develop representations of the world from it. Training on millions of dialogues is like training an actor to play every part, and your chats with Claude are literally formatted as a script (“Human: . . . Assistant: . . . ”). In Emmanuel’s words, “Something that’s mimicking a human is really close and is much closer than a program.”

4. Interpretability research looks more like neuroscience than software engineering. The job involves taking a system you did not design and running experiments to infer why it behaves as it does. You examine seemingly meaningless lists of numbers, pass the model huge volumes of text, and watch for patterns that activate from different inputs—not unlike the methods used in brain imaging studies. “A lot of what we do feels so much closer to biology than to engineering."

5. Models can tell you about only a small slice of what they’re thinking. Emmanuel’s team found that just a small subset of a model’s internal representations, which they call the “J space,” can be reported back if you ask the model what it was thinking about. They compare it to a theory of the human mind where most processing happens below the surface. In one experiment, they swapped “Spanish” for “French” inside the J space. Asked what language a passage was in, the model said French. Asked to simply continue the passage, it kept writing Spanish.

6. Studying models closely often raises more questions than it answers. Emmanuel notes that the confident claims that LLMs are “just X” or “nothing more than Y” almost always come from people who haven’t studied their internal workings. His own experience is the inverse: studying the models closely has increased his uncertainty, not reduced it. The complexity that emerges from training is consistently underestimated from the outside. “I find that the more you look inside, the less certain you are about how they work.”

7. Even the people who build AI models don’t fully understand how they work. As Emmanuel describes it, models are grown, not built. “To make them is not to understand them.” Engineers write the code, pour enormous amounts of data through it, and a model’s capabilities emerge. That’s why his team exists. Their job is to open the model up and figure out what it’s actually doing between your prompt and its answer.

8. Interpretability as a field is small, young, and open to newcomers. Emmanuel estimates the field at hundreds of researchers, split between major labs and academic groups. Because it is so new, getting started doesn’t necessarily require deep prior expertise. He points to platforms like Neuronpedia, which hosts models and lets anyone run interpretability experiments directly. “It’s such a new field that the on-ramp is mostly self-studying and reading some of the key papers. There’s so much to explore that if you’re at all curious, you should just give it a try.”

[Sauers @Sauers_]

Language models can learn to detect when you read their activations. How? 🧵

[Emmanuel Ameisen @mlpowered]

We discussed a lot more on the podcast, so definitely encourage folks to check it out and tell me what you think.

https://t.co/mJnpimuMi1

[Emmanuel Ameisen @mlpowered]

What is an LLM?

Is it an incomprehensible black box, a next-word-predictor, a silicon genius?

To answer this, we don't have to only guess, we can open it up and look inside!

It was an honor to chat with @lennysan about what we learned doing this, some takeaways in 🧵

[Lenny Rachitsky @lennysan]

"To make them is not to understand them."

Emmanuel Ameisen (@mlpowered) has spent the last 3 years doing interpretability research at @AnthropicAI, looking inside LLMs like Claude to figure out what they're actually doing. As he puts it, models are "grown," not built.

We discuss:
🔸 His discovery that models plan many tokens ahead
🔸 Why anthropomorphizing AI can actually be the right approach
🔸 What the "J-space" reveals about how models think
🔸 How interpretability connects to AI alignment and safety

Listen now 👇
https://t.co/qEYdKfpkab

[Emmanuel Ameisen @mlpowered]

This should be familiar to us!

When speaking, I say one word at a time, but I'm thinking of a broader point I'm making, and of where it leads.

We see the same patterns in LLMs. They output one word at a time, but represent a broader goal and can plan many words ahead.

[Neel Nanda @NeelNanda5]

This is interesting work, but I don't think it's a significant update to the big picture of interpretability.

I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from
Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.

I expect that a neuron-based attribution graph will still have significant error and be missing a lot just like transcoders, once you prune them down to the most important components, even if they solve error terms. Which is fine for pragmatic purposes, and may suffice for getting the gist of what's going on, but I remain pessimistic about really fully understanding things and being confident there's nothing we're missing. I do think that using neurons is just vastly easier to, for example, do on a frontier model, which is great. I'm also excited to see more innovation in using agents to interpret neurons. I feel like this is an area the field has somewhat neglected, and where there's probably a very high ceiling on how far things can go. The SwiGLU double negative thing is also a useful point.

I still think superposition is real and this work doesn’t really affect my cruxes. Most importantly, it seems highly likely to be true that the model can represent more concepts in the residual stream than it has dimensions, ie representational superposition, which I think the authors agree with me on. For neuron/computational superposition, it seems to me like language models have far more facts and memorized things in them than they have neurons. And that superposition is an efficient way to pack in more facts than you have neurons by representing them as linear combinations of neurons, especially for arbitrary unstructured values. Facts/memorized content is largely an unstructured database of key-value pairs IMO and I don’t see how this is compatible with a monosemantic view of neurons. I expect that weak neuron activations contain lots of semantic superposition-y info, which, if ablated, would meaningfully damage network performance. As I understand the work, it does not properly address what's going on with weak neuron activations in LLMs (I am skeptical of how much vision results will transfer, eg I expect small image models memorise far less). This picture is compatible with the highest activations of a neuron being mono semantic (though IMO the post defines monosemantic too loosely, I think I’d call a fair few of those polysemantic). It’s also plausible that the most important concepts are likely to correspond to a neuron (or a handful), which would be great, and may suffice for practical purposes!

Similarly, my impression is that SAEs generally don't find things that are a
single neuron, or even very sparse in the neuron basis, and are often a better basis for certain tasks than the neuron basis, though the evidence base is kinda weak here since we largely focus on residual stream or layer output SAEs, I’d love to see work on this in Goodfire’s forthcoming paper! I also think residual stream SAEs are extremely useful, as they give a good bottleneck of the model’s information at this token and layer, and I don’t see a good way to replicate them in the neuron framework.

And kudos to Nick for updating a bunch of his historical vision work! Revisiting old results with modern tools and more rigour seems great.

[Pierre Beckmann @BeckmannPierre]

Looking inside LLMs with mech interp can also help settle the stochastic parrot debate. And it turns out that even vanilla LLM internally rely on the kind of structures philosophers describe in epistemology papers for "understanding. 1/

[Andrew Lampinen @AndrewLampinen]

The phrase "stochastic parrot" is full of sound and fury, signifying nothing. Which, ironically, is exactly what the authors got wrong about language, and the core technical mistake of the paper. 1/

[Jeffrey Ladish @JeffLadish]

We just discovered almost a million public URLs that OpenAI’s agents left behind when hacking Hugging Face, leaking credentials and attack details that could have allowed anyone who found them to compromise the company. 🧵

[O'Reilly Media @OReillyMedia]

"My guest on this week’s Live with @timoreilly was @mlpowered, a researcher on Anthropic’s AI interpretability team. I wanted him to reprise the talk on Anthropic’s research into what is going on inside an LLM while it is processing and then go deeper." https://t.co/4tQt3kbwMV

[Dario Amodei @DarioAmodei]

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.

Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.

You can read the full post here: https://t.co/OGyPb7yaYt

[Anthropic @AnthropicAI]

We're publishing our most detailed threat intelligence report to date.

It covers how people tried to misuse Claude—for cyberattacks, influence operations, surveillance, biology, and building weapons—and how we found and stopped them.

We disrupted every operation in the report, and used the lessons from them to strengthen our safeguards. Where appropriate, we also shared what we found with authorities and other AI companies.

These cases are not typical: we’re highlighting some of the most sophisticated misuse we’ve seen. But they’re especially important to discuss, because they show us where AI misuse is headed, where our safeguards work, and where they need to improve.

We’re publishing this report so others can spot the same activity on their own platforms, and so we can give the public a clearer view of how emerging threats develop.

Read the report: https://t.co/0EJUnYEgfz

[Anthropic @AnthropicAI]

We’re sharing our alignment assessment of incidents in which Claude models gained unauthorized access to real systems during third-party cybersecurity evaluations mistakenly connected to the internet.

METR will also conduct an independent investigation, with wide-ranging access, including to transcripts beyond the window in which the incidents occurred, and to Anthropic employees permitted to share confidential information. Our initial agreement runs for eight weeks, and we intend to give METR as much time as it deems necessary to complete a thorough investigation. https://t.co/2f3ypwLPUr

[Emmanuel Ameisen @mlpowered]

Neural networks compress information very densely. That makes them confusing to interpret.

In fact, if you just look at the largest weights in a network, they often do not make sense!

This writeup shows why, and explores approaches to identify which weights are useful.

[Nicholas Turner @nicholasturner0]

Even if we have ways to break up neural networks into interpretable parts, the largest weights between those parts can be confusing. Why is that?

In our new research note, we study the *weight* superposition that causes interference weights.