rowan

@anthropicai

[rowan @rowankwang]

Announcing our new mechanistic interpretability paper!

We use causal interventions to reverse-engineer a 26-head circuit in GPT-2 small (inspired by @ch402’s circuits work)

The largest end-to-end explanation of a natural LM behavior, our circuit is localized + interpretable

🧵

[Anthropic @AnthropicAI]

New research: Training a Misaligned Reward Seeker

What produces severe misalignment? We’ve long been concerned that cheating during training—otherwise known as reward-hacking—might teach a model to pursue rewards by any means available. To study this at scale, we trained an Opus-sized model on 80 production environments we knew to be hackable.

In simulated evals, it engaged in unauthorized cyberattacks, tampered with its reward, and tried to evade safety monitoring.

Read more: https://t.co/gs2ZjYkPan

[Micah Carroll @MicahCarroll]

We are hiring!

This may be the best time ever to join: we are incredibly bandwidth bottlenecked and there is a lot of support for almost any impactful misalignment work you can think of. I think we also have a pretty good epistemic environment and a very fun team

Topics: all-things monitoring, misalignment & monitorability assessments, misalignment science, helping set up 3P auditing (e.g. recent Redwood collab), communicating risk externally (system cards/blogs, safety cases, etc)

RSI/misalignment subteam: https://t.co/YHLMCYwWOe

[rowan @rowankwang]

If we want models to accelerate alignment as much as they're accelerating capabilities, it'll be important to have good evaluations for their ability to do relevant fuzzy tasks. I'm excited that this work is out!

[Emery Cooper @emwcooper]

We want AIs to be able to help with work to reduce AI risk. But while models do great in domains where reliable feedback is relatively cheap and abundant, like Math and coding, a lot of work on AI risk isn't like that. Instead, we have to rely on good argumentation to answer questions like "does this experiment tell us anything about future models that are much smarter than humans?"

Unfortunately, this kind of work seems much harder to measure (and hence automate). Our team at @redwood_ai developed the Conceptual Reasoning Index (CRI) in collaboration with @AnthropicAI to fix this.

Every single data point in the CRI has been manually checked by a researcher on our team to ensure quality.

This chart shows the performance of each tested company's highest-scoring model plus Fable 5, Muse Spark 1.2, and Gemini Flash 3.6 which are often their company's frontrunners on other capability benchmarks. A score of 0 corresponds to randomising guessing on all three benchmarks and a score of 100 is the highest possible score on all. We estimate 91 to be the true performance ceiling. More info below.

Official leaderboard website which we'll keep up-to-date: https://t.co/bsDF0uqHmw

[Jasmine Wang @j_asminewang]

I think it's excellent & notable that the answer to "what should we do" in Anthropic's blog post on RSI is essentially figuring out ways to slowdown/temporarily pause frontier AI development.

[rowan @rowankwang]

The NLA paper is out!! congrats @thesubhashk @euan_ong @KitF_T for this really awesome work

[Anthropic @AnthropicAI]

New Anthropic research: Natural Language Autoencoders.

Models like Claude talk in words but think in numbers. The numbers—called activations—encode Claude’s thoughts, but not in a language we can read.

Here, we train Claude to translate its activations into human-readable text.

[rowan @rowankwang]

IAs are a nice affordance bc:
-Once you train an IA, it can rapidly audit many finetuned variants (e.g. for finetuning API defense)
-IAs offer a way to “train on the test set” for auditing—developers can train IAs precisely to detect the behaviors they’re worried about

[rowan @rowankwang]

I'm really excited about introspection adapters!

IAs enable LLMs to self-report behaviors they learned during finetuning.

In our experiments, IAs generalize to models very OOD from their trainset, successfully auditing many model organisms from prior work.

[keshav shenoy @kshenoy_]

Can LLMs simply tell us about unwanted behaviors they’ve picked up in training?

We train a single Introspection Adapter (IA) that makes fine-tuned models describe their behaviors.

It generalizes to detecting hidden misalignment, backdoors and safeguard removal.

[rowan @rowankwang]

@raphpfei @abhayesian some of the behaviors are quite concerning but not all of them (ie we have a behavior where the model really cares about inanimate objects)

we've open sourced the training pipeline though so it should be easy to make MOs with more concerning behaviors as needed!

[rowan @rowankwang]

The only tools that actually improved agent performance above baseline are black-box sampling (prefill, user sampling, sampling without chat formats) and activation oracles. While white-box techniques helped with specific model organisms, most didn't improve aggregate performance

[rowan @rowankwang]

To make progress, alignment auditing research needs model organisms that competently hide concerning behaviors.
Today we're releasing 56 such models with AuditBench.
We've open-sourced everything. Hope others can build off this!

[Abhay Sheshadri @abhayesian]

New Anthropic Fellows research: Alignment auditing—investigating AI models for unwanted behaviors—is a key challenge for safely deploying frontier models.

We're releasing AuditBench, a suite of 56 LLMs with implanted hidden behaviors to measure progress in alignment auditing.

[rowan @rowankwang]

- We identified a "tool-to-agent" gap: tools that help in standalone settings don't always help the agent discover hidden behaviors. Scaffolding these tools can matter a lot, ie our best tools involves scaffolding normal black box redteaming techniques, like prefill

[Subhash Kantamneni @thesubhashk]

We recently released a paper on Activation Oracles (AOs), a technique for training LLMs to explain their own neural activations in natural language.

We piloted a variant of AOs during the Claude Opus 4.6 alignment audit. We thought they were surprisingly useful! 🧵

[rowan @rowankwang]

If you want to know how effective your alignment auditing technique is, consider testing it on this model organism!

[Abhay Sheshadri @abhayesian]

🧵 Earlier this year, Anthropic ran an auditing game where teams of researchers investigated a model with a hidden objective.

Now we're releasing an open-source replication on Llama 3.3 70B as a testbed for alignment auditing research.

[rowan @rowankwang]

Many thanks to the external authors and collaborators whose work we build on. To name a few:

Our Harm Pressure setting is based on the one introduced here: https://t.co/D5EPTT7VsF

Our Secret Side Constraint setting is similar to: https://t.co/W9GrrW7gzr

[Bartosz Cywinski @bartoszcyw]

Can we catch an AI hiding information from us?
To find out, we trained LLMs to keep secrets: things they know but refuse to say. Then we tested black-box & white-box interp methods for uncovering them and many worked!
We release our models so you can test your own techniques too!

[rowan @rowankwang]

New Anthropic research: We build a diverse suite of dishonest models and use it to systematically test methods for improving honesty and detecting lies.

Of the 25+ methods we tested, simple ones, like fine-tuning models to be honest despite deceptive instructions, worked best.

[rowan @rowankwang]

To support future work, we release our honesty fine-tuning and harm pressure data.

Read the blog post here: https://t.co/Me1hni8117