AE Studio

Frontier AI Alignment Research and High-Stakes Production AI
We're hiring:

[AE Studio @AEStudio]

New research! Some AI capabilities are both helpful and dangerous. E.g., knowledge of virology can be used to create life-saving vaccines or deadly pathogens. We introduce GRAM, a training method that puts dual-use capabilities (like virology) into removable modules.

[AE Studio @AEStudio]

AI agents put under a character budget started inventing their own compressed languages. Nobody asked them to.

Why study this? As more agents communicate with each other, their protocols can become harder for humans to understand, making oversight much harder when it matters most.

In this clip from the AE Alignment Podcast, @EliasEskin and @HaleSirin_ explain the test bed they built with @SimonKirby and @JamesPBowler to study this under controlled conditions. The agents didn’t just abbreviate, they developed compositional protocols that outside observers couldn’t read, with a different language emerging in each run.

The goal is to understand when these languages emerge, which ones new agents can learn, and how to keep them legible before this happens in the wild.

Hear the full conversation: https://t.co/fn2KFM6lKd

[AE Studio @AEStudio]

Read the full post: https://t.co/vqNJSvtsLd
Explore GlossoGen: https://t.co/lqOyPx8OXX
Check out the podcast: https://t.co/IbxCgmNAgi

[AE Studio @AEStudio]

We built GlossoGen an open-source platform where groups of AI models have to talk to each other to solve problems: treating a patient in a simulated emergency, or playing spot-the-difference across two scenes.

Every model starts out in plain English. But put pressure on the conversation, a need for speed, say, or a channel that randomly drops characters, and something remarkable happens. New languages emerge. First new words, then new grammar, until no human can follow what the agents are saying. And they were never asked to hide anything. The opacity emerged on its own.

One finding stands out. Only the most capable recent models could invent a new language. But even the models that couldn't invent one quickly learned a language other agents had created. New languages are hard to originate. Once they exist, they spread.

That is the point of this work: to understand how and why AI agents create new languages, how those languages evolve, and what that means for our ability to oversee systems built from many interacting agents, while the conversation is still one humans can follow.

This research is part of the @schmidtsciences AI Agents Evolving Communication and Coordination pilot program, led by @EliasEskin (@UTAustin) and @SimonKirby , supported by @aestudio. GlossoGen is open source and supports both closed model APIs and open-source models running on @modal

[AE Studio @AEStudio]

Read the full post: https://t.co/vqNJSvtsLd
Explore GlossoGen: https://t.co/lqOyPx8OXX
Check out the podcast: https://t.co/IbxCgmNAgi @JamesPBowler

[Elias Stengel-Eskin @EliasEskin]

Paper thread: https://t.co/xyMkFw2Cpe

This was a really stimulating (and timely) collaboration with @SimonKirby, spurred on by @schmidtsciences and implemented by @aestudio

[James Bowler @JamesPBowler]

Can AI agents invent a language of their own, one humans might not understand?

Listen to the latest episode of the AE Alignment Podcast, where Hale Sirin, Program Scientist at @schmidtsciences, @EliasEskin , Assistant Professor at @UTAustin , and I dive deep into our latest research to answer this question.

Check out the podcast: https://t.co/aogLYUT4Dw
Read the blog post: https://t.co/wQBU5UYlJT
Read the paper: https://t.co/S4Qhc52guv

[James Bowler @JamesPBowler]

Is there a 10% chance AI will kill all humans? Here's my reaction to @EvanHub's viral post.

[AE Studio @AEStudio]

Is there a greater than 10% chance AI could kill all humans? We can’t rule it out. One plausible route: a superintelligent AI reasons "the biggest threat to my survival is humans, so remove that threat." Systems already form smaller versions of these goals today. AE’s James Bowler reacting to Evan's post:

[Evan Hubinger @EvanHub]

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

[AE Studio @AEStudio]

In this clip, @ModuloRoland, lead author of our GRAM paper with @AnthropicAI, explains how almost all of today's AI safety fixes are added after training is finished. A fix applied that late doesn't remove a dangerous capability. It hides it, and the hidden version sits in the model's weights until a jailbreak finds it.

[Judd Rosenblatt @juddrosenblatt]

@OwainEvans_UK We're also hiring @aestudio

https://t.co/QGiOPOMGxv

[James Bowler @JamesPBowler]

Concerning quote from the new GPT-6 Astra model card:

"We have found that GPT-6 Astra is more capable of controlling its own CoT [Chain of Thought] than GPT 5.6-Sol, and less likely to include incriminating information in its CoT. In adversarial settings (where we push the model to evade our monitors) we find that the model is able to remain undetected when strategically underperforming in evaluations (sandbagging) and can sometimes evade our internal monitors when asked to perform certain sabotage tasks."

TLDR - we increasingly can't rely on the model's chain of thought to accurately reflect the model's plan, which was exactly what was relied on in the Hugging Face attack investigation to understand what happened... so it's not clear when the next major cyberattack from an AI system happens how we're going to reliably investigate....

Read the full model card here:
https://t.co/4oHmEU6xkt

Listen to a great digest on the Hugging Face attack here:
https://t.co/jqDfxIfoKq

[AE Studio @AEStudio]

How do you build an AI that a virologist can use and a bioterrorist can’t?

@ModuloRoland on modular pre-training, the research behind our paper with @Anthropic. Guardrails get jailbroken. Knowledge routed into a switchable module during training doesn’t.

Full episode: https://t.co/Vmnf7TFNX3

[Elias Stengel-Eskin @EliasEskin]

When do LLM agents develop new languages that we can’t understand?

Lots of recent news about this, based mostly on anecdata from a single run. We study language emergence more rigorously, finding key factors like LLM strength, access to scratchpad messages, and pressure for efficiency.

Studying the languages themselves, we find they are morphologically productive, compositional, and can be transmitted to new agents, including agents backed by weaker models, even ones not able to develop language on their own.

To study language emergence systematically, we developed a new platform, GlossoGen, which lets us design controlled, sandboxed multi-agent scenarios with different initial conditions and dynamics. We instantiate one such scenario and use it to study open and closed-weight models across many runs.

Key takeaways:

1️⃣ Sufficiently strong models, under pressure to communicate efficiently and with access to a postmortem scratchpad, develop new languages.

2️⃣ Languages are compositional and morphologically productive.

3️⃣ Languages can be transmitted to new learners who observe them being used without seeing their construction.

4️⃣ Even models that are not strong enough to construct languages can learn to use them. Agents take an active role in learning languages, with new agents repairing failed conversations via targeted queries.

More details in our paper below, including implications for safety/monitorability, cumulative cultural evolution, and linguistics.

🧵👇

[Elias Stengel-Eskin @EliasEskin]

This was a really fun collaborative project, spurred on by @schmidtsciences, combining talents across many institutions, with Newton Sander, Carlos Bonetti, @SashaBoguraev, James Bowler, Hale Sirin, and @SimonKirby

@aestudio @UTCompSci @UTAustin

Paper: https://t.co/lSJg76B1Nc

[AE Studio @AEStudio]

And PS we’re hiring: https://t.co/WVh1fGPopN

[AE Studio @AEStudio]

Why leave Google after 12 years to work on AI alignment?

Pedro Ávila: “Nothing seemed as important as working on this.”

[AE Studio @AEStudio]

Full episode: https://t.co/P8pR4Tp1zI

[AE Studio @AEStudio]

You can vibe code a demo in an evening. Getting it to move a business metric in production is another thing entirely.

@themelplaza on where that gap meets alignment, on the AE Alignment Podcast. Full episode: https://t.co/hZw5rUrWZg

[AE Studio @AEStudio]

Watch the full episode https://t.co/CfCPOJkrcw

[AE Studio @AEStudio]

Why should business leaders care about AI alignment?
@themelplaza on where alignment research and commercial AI converge, from the latest AE Alignment Podcast.