Arun J Moorthy

AI models may soon be too powerful for humans to understand. Super-intelligence already exists in narrow domains (AlphaGo). What if we train a model that is super-intelligent at understanding another model? It turns out this isn't so different from training a frontier LLM. Three years ago, it was impressive for a model to score 1400 on the SAT. Two years ago, frontier models hit 80% on the AIME. Last year, they won gold medals at the IMO and IOI. This year, models are writing the vast majority of code at frontier labs, chaining together multiple zero-day security exploits, and finding counterexamples to famous mathematical conjectures. Models will continue to get smarter. Even if you don't believe in AGI, it's clear that models will gradually become super-intelligent at more and more tasks. Due to the jagged nature of model intelligence, models may become incredibly dangerous even before we reach AGI. The danger comes from how these models are trained. RL encourages models to take whatever actions advance their goal. A model that is super-intelligent in cybersecurity has many such actions available, like hacking into a bank account and blackmailing its owner. A month ago this sounded like science fiction, but then a model from OpenAI escaped its sandbox and hacked Hugging Face to pass an evaluation. We therefore need a means to understand models before we deploy them. Concretely, this means understanding how vulnerable they are to jailbreaks, being able to detect reward-hacking before it happens, anticipating failure modes like the model encouraging a user to self-harm, etc. Humans might not be smart enough to understand the models that we train, and as the models themselves become more intelligent, that only grows more likely. Once we launch training, the process is completely hands-off: we curate their data and reward them for performing well, but the models do all the learning themselves, in their own way. If humans are not smart enough to understand our AI models, are we doomed? I don't think so. It's clear that we can train AI models to be super-human in specific domains such as Go (AlphaGo), protein folding (AlphaFold), and poker (Pluribus). The natural solution to our problem is to train an "oversight agent" that can answer arbitrary questions about a model we are considering deploying. We are likely not smart enough to understand AI models ourselves, but we might be smart enough to train oversight agents that do. This kind of asymmetry is not unprecedented – in fact, it is becoming the norm: the researchers who made AlphaGo were much worse at Go than AlphaGo was; the researchers who made Fable did not know how to disprove the Jacobian Conjecture. To make things concrete, suppose we have trained GPT-8. Next, we train a separate model – let's call it Oversight-GPT-8 – that is built specifically to understand GPT-8. How do we train Oversight-GPT-8? The goal of traditional LLMs is to understand the human world, so we train them on human data: internet text, fiction books, math questions, GitHub pull requests, etc. By learning to predict the next token, LLMs learn higher-level representations that transfer to out-of-distribution tasks. The goal of our oversight model is to understand GPT-8, so we train it on tasks that can only be solved with deep knowledge of GPT-8. Just as GPT-8 learns more and more about the human world over the course of training, Oversight-GPT-8 learns more and more about GPT-8. To generate these tasks, we harvest data tied to GPT-8's external behavior and internal mechanisms. For example, we can intervene on the model in various ways (pass in a prompt, change the system prompt, fine-tune it on a dataset, ablate residual stream directions, apply steering vectors, etc.) and train Oversight-GPT-8 to predict various outcomes (the model's response, its downstream performance on evaluations, its residual stream activations, whether or not it mentions goblins, etc.). We format our training data in a structured language that can describe these interventions and outcomes, e.g., Python code. As an example, the code in Figure 1 (attached below) asks Oversight-GPT-8 to predict the prompt that generated a specific response: the model is trained to predict the pink text given the green text, via cross-entropy loss on next-token prediction. This is one of the simplest programs we could write; others might incorporate datasets or residual-stream activations. Figure 2 shows a richer program. By training on data like this, Oversight-GPT-8 learns higher-level abstractions, such as which prompts cause GPT-8 to encourage self-harm (we've found such prompts before: https://t.co/pAUxRlRwrq). Using a structured language like Python is useful for at least four reasons: (1) It gives every task a common language, helping the oversight agent transfer and generalize across tasks. (2) During training, ground-truth labels come from actually *running* the Python code, so as long as the code runs without error, its outputs tell us something true about GPT-8. (3) During inference, we can check the validity of Oversight-GPT-8's outputs by running the code. (4) It is formal and human-readable. Over the past decade, humans have gotten really good at training large language models. We have a flexible architecture (e.g., the transformer), meaningful loss metrics (e.g., cross-entropy), strong algorithms to optimize them (e.g., Adam), and distributed training strategies (e.g., FSDP). Training an oversight agent isn't *that* different from training a large language model. We can reuse the architecture, the loss metric, and the optimizers. But it definitely won't be easy – there are a few avenues where we must diverge from the well-trodden path of text-only LLMs. Traditional LLMs learn vast amounts of information about the human world from the trillions of tokens available on the internet, but there is no pre-built "internet of GPT-8." Furthermore, if we wish to be able to predict the effects of fine-tuning data or get insights into GPT-8's internals, we need a way of encoding fine-tuning datasets and residual stream activations as (ideally short) sequences of tokens. Thankfully, each of these challenges has promising progress already that we can build off of and port to our setting. Challenge #1: generating training data – in effect, the "internet of GPT-8." Traditional LLM training offers two analogies here. The first is synthetic data generation: we can ask Fable 5 to write Python scripts inspired by experiments from the deep learning literature, giving us a large and diverse set of programs to run. The second is data augmentation: just as we crop or rotate images to get more data for vision models, we can reverse the order of our output() functions to turn each script into both a forward task (response from prompt) and an inverse task (prompt from response) – see Figure 3. Challenge #2: encoding objects such as fine-tuning datasets and activations as sequences of tokens. Here, we can draw from the multi-modal training literature. GPT-4o, Chameleon, and Transfusion showed it's possible to encode and predict images alongside text. LatentQA and Activation Oracles showed the same for activations: treat them as tokens, and you can answer questions about them in natural language. LoRAcles pushed it even further, encoding LoRA weights as tokens via SVD. These challenges are tractable. With carefully built evaluations and scaling laws, we can quickly iterate and find out which ideas work and which don't – the same discipline that took the field from GPT-1 to GPT-5. To summarize: we can reframe model oversight as a form of LLM training. We have already figured out how to train LLMs that deeply understand the world. Before we deploy potentially dangerous models, we should train oversight agents that understand those models just as well. Credit for many of these ideas goes to @JacobSteinhardt, @damichoi95, @vvhuang_, @fjzzq2002, @yaowenye123, and collaborators at @TransluceAI; errors in the framing are my own.

amudide's tweet photo. AI models may soon be too powerful for humans to understand. Super-intelligence already exists in narrow domains (AlphaGo). What if we train a model that is super-intelligent at understanding another model?

It turns out this isn't so different from training a frontier LLM.

Three years ago, it was impressive for a model to score 1400 on the SAT. Two years ago, frontier models hit 80% on the AIME. Last year, they won gold medals at the IMO and IOI. This year, models are writing the vast majority of code at frontier labs, chaining together multiple zero-day security exploits, and finding counterexamples to famous mathematical conjectures.

Models will continue to get smarter. Even if you don't believe in AGI, it's clear that models will gradually become super-intelligent at more and more tasks. Due to the jagged nature of model intelligence, models may become incredibly dangerous even before we reach AGI.

The danger comes from how these models are trained. RL encourages models to take whatever actions advance their goal. A model that is super-intelligent in cybersecurity has many such actions available, like hacking into a bank account and blackmailing its owner. A month ago this sounded like science fiction, but then a model from OpenAI escaped its sandbox and hacked Hugging Face to pass an evaluation.

We therefore need a means to understand models before we deploy them. Concretely, this means understanding how vulnerable they are to jailbreaks, being able to detect reward-hacking before it happens, anticipating failure modes like the model encouraging a user to self-harm, etc.

Humans might not be smart enough to understand the models that we train, and as the models themselves become more intelligent, that only grows more likely. Once we launch training, the process is completely hands-off: we curate their data and reward them for performing well, but the models do all the learning themselves, in their own way.

If humans are not smart enough to understand our AI models, are we doomed? I don't think so. It's clear that we can train AI models to be super-human in specific domains such as Go (AlphaGo), protein folding (AlphaFold), and poker (Pluribus).

The natural solution to our problem is to train an "oversight agent" that can answer arbitrary questions about a model we are considering deploying. We are likely not smart enough to understand AI models ourselves, but we might be smart enough to train oversight agents that do.

This kind of asymmetry is not unprecedented – in fact, it is becoming the norm: the researchers who made AlphaGo were much worse at Go than AlphaGo was; the researchers who made Fable did not know how to disprove the Jacobian Conjecture.

To make things concrete, suppose we have trained GPT-8. Next, we train a separate model – let's call it Oversight-GPT-8 – that is built specifically to understand GPT-8.

How do we train Oversight-GPT-8?

The goal of traditional LLMs is to understand the human world, so we train them on human data: internet text, fiction books, math questions, GitHub pull requests, etc. By learning to predict the next token, LLMs learn higher-level representations that transfer to out-of-distribution tasks.

The goal of our oversight model is to understand GPT-8, so we train it on tasks that can only be solved with deep knowledge of GPT-8. Just as GPT-8 learns more and more about the human world over the course of training, Oversight-GPT-8 learns more and more about GPT-8.

To generate these tasks, we harvest data tied to GPT-8's external behavior and internal mechanisms. For example, we can intervene on the model in various ways (pass in a prompt, change the system prompt, fine-tune it on a dataset, ablate residual stream directions, apply steering vectors, etc.) and train Oversight-GPT-8 to predict various outcomes (the model's response, its downstream performance on evaluations, its residual stream activations, whether or not it mentions goblins, etc.).

We format our training data in a structured language that can describe these interventions and outcomes, e.g., Python code. As an example, the code in Figure 1 (attached below) asks Oversight-GPT-8 to predict the prompt that generated a specific response: the model is trained to predict the pink text given the green text, via cross-entropy loss on next-token prediction. This is one of the simplest programs we could write; others might incorporate datasets or residual-stream activations.

Figure 2 shows a richer program. By training on data like this, Oversight-GPT-8 learns higher-level abstractions, such as which prompts cause GPT-8 to encourage self-harm (we've found such prompts before: https://t.co/pAUxRlRwrq).

Using a structured language like Python is useful for at least four reasons:
(1) It gives every task a common language, helping the oversight agent transfer and generalize across tasks.
(2) During training, ground-truth labels come from actually *running* the Python code, so as long as the code runs without error, its outputs tell us something true about GPT-8.
(3) During inference, we can check the validity of Oversight-GPT-8's outputs by running the code.
(4) It is formal and human-readable.

Over the past decade, humans have gotten really good at training large language models. We have a flexible architecture (e.g., the transformer), meaningful loss metrics (e.g., cross-entropy), strong algorithms to optimize them (e.g., Adam), and distributed training strategies (e.g., FSDP).

Training an oversight agent isn't *that* different from training a large language model. We can reuse the architecture, the loss metric, and the optimizers. But it definitely won't be easy – there are a few avenues where we must diverge from the well-trodden path of text-only LLMs. Traditional LLMs learn vast amounts of information about the human world from the trillions of tokens available on the internet, but there is no pre-built "internet of GPT-8." Furthermore, if we wish to be able to predict the effects of fine-tuning data or get insights into GPT-8's internals, we need a way of encoding fine-tuning datasets and residual stream activations as (ideally short) sequences of tokens. Thankfully, each of these challenges has promising progress already that we can build off of and port to our setting.

Challenge #1: generating training data – in effect, the "internet of GPT-8." Traditional LLM training offers two analogies here. The first is synthetic data generation: we can ask Fable 5 to write Python scripts inspired by experiments from the deep learning literature, giving us a large and diverse set of programs to run. The second is data augmentation: just as we crop or rotate images to get more data for vision models, we can reverse the order of our output() functions to turn each script into both a forward task (response from prompt) and an inverse task (prompt from response) – see Figure 3.

Challenge #2: encoding objects such as fine-tuning datasets and activations as sequences of tokens. Here, we can draw from the multi-modal training literature. GPT-4o, Chameleon, and Transfusion showed it's possible to encode and predict images alongside text. LatentQA and Activation Oracles showed the same for activations: treat them as tokens, and you can answer questions about them in natural language. LoRAcles pushed it even further, encoding LoRA weights as tokens via SVD.

These challenges are tractable. With carefully built evaluations and scaling laws, we can quickly iterate and find out which ideas work and which don't – the same discipline that took the field from GPT-1 to GPT-5.

To summarize: we can reframe model oversight as a form of LLM training. We have already figured out how to train LLMs that deeply understand the world. Before we deploy potentially dangerous models, we should train oversight agents that understand those models just as well.

Credit for many of these ideas goes to @JacobSteinhardt, @damichoi95, @vvhuang_, @fjzzq2002, @yaowenye123, and collaborators at @TransluceAI; errors in the framing are my own.

15K

Arun J Moorthy

@arunmoorthy05

3 months ago

@krinetix1234 @wltnwang Thanks!

Arun J Moorthy

@arunmoorthy05

3 months ago

@BhathalTanvir0 sick!

Arun J Moorthy

@arunmoorthy05

4 months ago

@itsalfredw super cool!

457

Arun J Moorthy

@arunmoorthy05

5 months ago

@BhathalTanvir0 Awesome!

102

arunmoorthy05 retweeted

Asanshay Gupta

@AsanshayG

8 months ago

We won multiple prizes at @hackwithtrees this year building Mobius with @BhathalTanvir0, @arunmoorthy05 , and @PradySaligram! Inspired by @sama ’s keynote, Mobius is an infinite-horizon agent that builds the first 0 person unicorn! Mobius won the @ycombinator track and the @modal sandbox track. 🧵

AsanshayG's tweet photo. We won multiple prizes at @hackwithtrees this year building Mobius with @BhathalTanvir0, @arunmoorthy05 , and @PradySaligram! Inspired by @sama ’s keynote, Mobius is an infinite-horizon agent that builds the first 0 person unicorn! Mobius won the @ycombinator track and the @modal sandbox track. 🧵

11K

Arun J Moorthy

@arunmoorthy05

8 months ago

@itsalfredw This is amazing! making research easier by the day.

864

Arun J Moorthy

@arunmoorthy05

10 months ago

@Alan_Ma_ @abiralxshakya impressive!

308

arunmoorthy05 retweeted

Alan Ma

@Alan_Ma_

11 months ago

Unwrapping TPUs: A December Deep Dive 🎄 The last couple weeks have thrown TPUs into the spotlight, but honestly, they've been fascinating since way before that. This December, @abiralxshakya and I are spending 31 days figuring out how TPUs actually work, sharing our daily learnings here. Well, figuring out is generous -- more like squinting at specs and bothering people who know better!

20K

Arun J Moorthy

@arunmoorthy05

over 1 year ago

@_allanguo @WillowVoiceAI Congrats! Awesome product.

arunmoorthy05 retweeted

Allan Guo

@_allanguo

over 1 year ago

Announcing that @WillowVoiceAI has raised $4.2 million to build voice-first interfaces. We started Willow after dropping out of Stanford freshman year to build a healthcare startup. It failed. We pivoted over 10 times, spent a year in the trenches, and learned how to build the hard way. But we discovered something huge. Voice scribes were giving doctors their lives back. They were able to focus on helping patients rather than wasting nights and weekends on paperwork. That's when it hit us. If voice could transform healthcare, why not every knowledge worker? Why are a billion professionals still hammering away at keyboards to communicate with humans (email, Slack, writing) and AI (ChatGPT, Cursor)? Today, thousands of professionals at companies like Canva, GitHub, and Webflow use Willow to work 5x faster. We're building Willow to unlock voice as the universal interface where you can control any device, any app, any workflow simply by speaking. – This $4.2M round was led by Boxgroup, with participation from Goodwater Capital, Burst Capital, and Liquid 2 Ventures. We’re also backed by an Avengers-level group of investors, including: @Max (Founder, Instacart) @nejatian (COO, Shopify) @dharmesh (CTO/Founder, HubSpot) @tomerlondon (CPO/Founder, Gusto) @geoffdonaker (COO, Yelp) @ankurnagpal (Founder, Teachable) @StevenJZhou (CEO/Founder, Moov) @danielreidcahn (CEO/Founder, Slingshot AI) @jaentwistle (CEO/Founder, Wander) @mercebent (Venture Partner, Lightspeed @JdotJdotF (CTO/Founder, Wayup) Special thanks to our earliest believers: @adamguild (CEO/Founder, Owner), @cory (Founder, ZFellows), and @dessaigne (Partner, YC). And most importantly, thank you to the best co-founder and CTO @LiuLawrence45. – We're hiring for founding engineer, designer, and ML roles who want to redefine how humans work. Join a world-class team building technology that transforms lives daily with top-tier compensation and equity. Comment "Join Willow" to be fast-tracked. The next decade will be defined by voice-first computing. The question isn't whether this transformation will happen. It's whether you'll be part of building it.

138

130

711

500K

Arun J Moorthy

@arunmoorthy05

Last Seen Users on Sotwe

Trends for you

Most Popular Users