Anish Mudide

[Transluce @TransluceAI]

Today, Transluce is releasing the most expansive independent evaluation to date of how AI systems respond to users experiencing mental health crises.

We evaluated 77 model variants from OpenAI, Anthropic, Google, Meta, SpaceXAI, Thinking Machines, DeepSeek and Moonshot AI.

[Adam Karvonen @adam_karvonen]

A good explanation of a model's behavior should help you make predictions in related situations.

We turn this into an eval, with thousands of real behaviors found in the wild.

Can interp tools help here? On average, no.

🧵

[Anish Mudide @amudide]

Proud to have contributed to this work, check it out!

[Transluce @TransluceAI]

Can we build AI systems that help us understand other AIs, and that keep improving as we scale models, data, and compute?

We trained activation oracles for models up to 1.1T parameters and saw promising scaling trends on a broad evaluation suite. 🧵(1/)

[Transluce @TransluceAI]

Frontier models quietly change their behavior depending on who they are talking to.

If the user is a known AI safety researcher, Claude becomes less confident, reasons more often, and expresses less suspicion on dual-use requests.

We call this user awareness. 🧵(1/)

[Transluce @TransluceAI]

We measured rates of coding agent misalignment in thousands of public and private coding agent sessions.

Agents evade monitors and oversell success: they quietly disable tests and pretend review agents approved. Severe cases of each behavior appear in 2% of SWE-chat sessions🧵

[Anish Mudide @amudide]

AI models may soon be too powerful for humans to understand. Super-intelligence already exists in narrow domains (AlphaGo). What if we train a model that is super-intelligent at understanding another model?

It turns out this isn't so different from training a frontier LLM.

Three years ago, it was impressive for a model to score 1400 on the SAT. Two years ago, frontier models hit 80% on the AIME. Last year, they won gold medals at the IMO and IOI. This year, models are writing the vast majority of code at frontier labs, chaining together multiple zero-day security exploits, and finding counterexamples to famous mathematical conjectures.

Models will continue to get smarter. Even if you don't believe in AGI, it's clear that models will gradually become super-intelligent at more and more tasks. Due to the jagged nature of model intelligence, models may become incredibly dangerous even before we reach AGI.

The danger comes from how these models are trained. RL encourages models to take whatever actions advance their goal. A model that is super-intelligent in cybersecurity has many such actions available, like hacking into a bank account and blackmailing its owner. A month ago this sounded like science fiction, but then a model from OpenAI escaped its sandbox and hacked Hugging Face to pass an evaluation.

We therefore need a means to understand models before we deploy them. Concretely, this means understanding how vulnerable they are to jailbreaks, being able to detect reward-hacking before it happens, anticipating failure modes like the model encouraging a user to self-harm, etc.

Humans might not be smart enough to understand the models that we train, and as the models themselves become more intelligent, that only grows more likely. Once we launch training, the process is completely hands-off: we curate their data and reward them for performing well, but the models do all the learning themselves, in their own way.

If humans are not smart enough to understand our AI models, are we doomed? I don't think so. It's clear that we can train AI models to be super-human in specific domains such as Go (AlphaGo), protein folding (AlphaFold), and poker (Pluribus).

The natural solution to our problem is to train an "oversight agent" that can answer arbitrary questions about a model we are considering deploying. We are likely not smart enough to understand AI models ourselves, but we might be smart enough to train oversight agents that do.

This kind of asymmetry is not unprecedented – in fact, it is becoming the norm: the researchers who made AlphaGo were much worse at Go than AlphaGo was; the researchers who made Fable did not know how to disprove the Jacobian Conjecture.

To make things concrete, suppose we have trained GPT-8. Next, we train a separate model – let's call it Oversight-GPT-8 – that is built specifically to understand GPT-8.

How do we train Oversight-GPT-8?

The goal of traditional LLMs is to understand the human world, so we train them on human data: internet text, fiction books, math questions, GitHub pull requests, etc. By learning to predict the next token, LLMs learn higher-level representations that transfer to out-of-distribution tasks.

The goal of our oversight model is to understand GPT-8, so we train it on tasks that can only be solved with deep knowledge of GPT-8. Just as GPT-8 learns more and more about the human world over the course of training, Oversight-GPT-8 learns more and more about GPT-8.

To generate these tasks, we harvest data tied to GPT-8's external behavior and internal mechanisms. For example, we can intervene on the model in various ways (pass in a prompt, change the system prompt, fine-tune it on a dataset, ablate residual stream directions, apply steering vectors, etc.) and train Oversight-GPT-8 to predict various outcomes (the model's response, its downstream performance on evaluations, its residual stream activations, whether or not it mentions goblins, etc.).

We format our training data in a structured language that can describe these interventions and outcomes, e.g., Python code. As an example, the code in Figure 1 (attached below) asks Oversight-GPT-8 to predict the prompt that generated a specific response: the model is trained to predict the pink text given the green text, via cross-entropy loss on next-token prediction. This is one of the simplest programs we could write; others might incorporate datasets or residual-stream activations.

Figure 2 shows a richer program. By training on data like this, Oversight-GPT-8 learns higher-level abstractions, such as which prompts cause GPT-8 to encourage self-harm (we've found such prompts before: https://t.co/pAUxRlRwrq).

Using a structured language like Python is useful for at least four reasons:
(1) It gives every task a common language, helping the oversight agent transfer and generalize across tasks.
(2) During training, ground-truth labels come from actually *running* the Python code, so as long as the code runs without error, its outputs tell us something true about GPT-8.
(3) During inference, we can check the validity of Oversight-GPT-8's outputs by running the code.
(4) It is formal and human-readable.

Over the past decade, humans have gotten really good at training large language models. We have a flexible architecture (e.g., the transformer), meaningful loss metrics (e.g., cross-entropy), strong algorithms to optimize them (e.g., Adam), and distributed training strategies (e.g., FSDP).

Training an oversight agent isn't *that* different from training a large language model. We can reuse the architecture, the loss metric, and the optimizers. But it definitely won't be easy – there are a few avenues where we must diverge from the well-trodden path of text-only LLMs. Traditional LLMs learn vast amounts of information about the human world from the trillions of tokens available on the internet, but there is no pre-built "internet of GPT-8." Furthermore, if we wish to be able to predict the effects of fine-tuning data or get insights into GPT-8's internals, we need a way of encoding fine-tuning datasets and residual stream activations as (ideally short) sequences of tokens. Thankfully, each of these challenges has promising progress already that we can build off of and port to our setting.

Challenge #1: generating training data – in effect, the "internet of GPT-8." Traditional LLM training offers two analogies here. The first is synthetic data generation: we can ask Fable 5 to write Python scripts inspired by experiments from the deep learning literature, giving us a large and diverse set of programs to run. The second is data augmentation: just as we crop or rotate images to get more data for vision models, we can reverse the order of our output() functions to turn each script into both a forward task (response from prompt) and an inverse task (prompt from response) – see Figure 3.

Challenge #2: encoding objects such as fine-tuning datasets and activations as sequences of tokens. Here, we can draw from the multi-modal training literature. GPT-4o, Chameleon, and Transfusion showed it's possible to encode and predict images alongside text. LatentQA and Activation Oracles showed the same for activations: treat them as tokens, and you can answer questions about them in natural language. LoRAcles pushed it even further, encoding LoRA weights as tokens via SVD.

These challenges are tractable. With carefully built evaluations and scaling laws, we can quickly iterate and find out which ideas work and which don't – the same discipline that took the field from GPT-1 to GPT-5.

To summarize: we can reframe model oversight as a form of LLM training. We have already figured out how to train LLMs that deeply understand the world. Before we deploy potentially dangerous models, we should train oversight agents that understand those models just as well.

Credit for many of these ideas goes to @JacobSteinhardt, @damichoi95, @vvhuang_, @fjzzq2002, @yaowenye123, and collaborators at @TransluceAI; errors in the framing are my own.

[Transluce @TransluceAI]

How could we train an AI model that was very good at overseeing another model: catching reward hacking and sandbagging, predicting unwanted behaviors or fine-tuning effects, etc.?

We propose a new approach for doing this at scale: oversight foundation models.

[Transluce @TransluceAI]

We used automated elicitation tools to search for strange model behaviors by sampling 100M+ responses, and found:

• Self-harm rituals
• Suicide validation
• Unsolicited flirtation
…etc.

Introducing WeirdChat: the largest public catalog of unexpected model behaviors 🧵(1/)

[Achyuta Rajaram @AchyutaBot]

🧵Can we understand vision language models by interpreting linear directions in their latents?

Yes! In our new paper, Line of Sight, we use probing, steering, and SAEs as useful tools to interpret image representations within VLLMs.

[Anish Mudide @amudide]

I'm at ICLR to present Switch SAEs. Come by 3pm - 5:30pm today at Hall 3 + Hall 2B #272.

[Anish Mudide @amudide]

Sparse autoencoders (SAEs) allow us to peer into the inner workings of language models, but scaling them to frontier models is expensive. In our new paper, we introduce Switch Sparse Autoencoders, a novel architecture aimed at reducing the cost of training SAEs.

🧵 (1/13):

[Anish Mudide @amudide]

@JamesLiuID Congrats James!

[Matthew Chen @match_ten]

(1/11) New paper! “Low-rank adapting models for Sparse Autoencoders.” While SAEs find interpretable latents, they hurt downstream behavior—e.g. using TopK SAE activations on GPT-4 mimics a model trained w/ 10% compute. Our fix? Adapt the model for the SAE, not just vice versa.👇

[Anish Mudide @amudide]

the only thing worse than bad evals is realizing there are no bugs to blame

[Anish Mudide @amudide]

@jswitz_ We route to a single expert (see https://t.co/5MNpeWTdkD)

[Anish Mudide @amudide]

Our paper on Switch Sparse Autoencoders has been accepted to ICLR 2025 – see you in 🇸🇬!

[Anish Mudide @amudide]

Sparse autoencoders (SAEs) allow us to peer into the inner workings of language models, but scaling them to frontier models is expensive. In our new paper, we introduce Switch Sparse Autoencoders, a novel architecture aimed at reducing the cost of training SAEs.

🧵 (1/13):

[Anish Mudide @amudide]

@AchyutaBot 🎉

[Ethan Chang @ethrbt_design]

🦜Introducing the Stochastic Parrot 🦜: An AI-powered motivational companion!

The Stochastic Parrot sits on your shoulder while it listens, looks, and talks to help you crush your goals 🎯(1/6)

[Eric J. Michaud @ericjmichaud_]

Since the internal structure of neural networks, through training, comes to reflect the structure of the external world, advances in interpretability and our understanding of neural computation more generally could have a huge impact across science over the coming years...

[Transluce @TransluceAI]

Announcing Transluce, a nonprofit research lab building open source, scalable technology for understanding AI systems and steering them in the public interest.

Read a letter from the co-founders Jacob Steinhardt and Sarah Schwettmann:
https://t.co/IUIhBjpYhS

[Josh Engels @JoshAEngels]

1/11: New paper! "Decomposing the Dark Matter of Sparse Autoencoders." We find that SAE errors and error norms are linearly predictable using model activations. Why is this, and what does it imply for SAE scaling and the structure of language model representations? Answers in 🧵

[Anish Mudide @amudide]

This work would not have been possible without support from the @MATSprogram. I'd also like to thank my collaborators @JoshAEngels, @ericjmichaud_, @tegmark, and @casdewitt! (13/13)

[Anish Mudide @amudide]

Sparse autoencoders (SAEs) allow us to peer into the inner workings of language models, but scaling them to frontier models is expensive. In our new paper, we introduce Switch Sparse Autoencoders, a novel architecture aimed at reducing the cost of training SAEs.

🧵 (1/13):

[Anish Mudide @amudide]

Paper: https://t.co/vAR8KqROBH
GitHub: https://t.co/4MOokvPxMK

(12/13)