Owen

MATS

[Owen @owen__terry]

@PeterNutter1 @JSchaeff3r Wow interesting

[Owen @owen__terry]

@shiv_kampani @VishnuSampathk1 @ycombinator 🙌

[Owen @owen__terry]

@KamacheeMax Huge

[Owen @owen__terry]

@precognita Very interesting

[Owen @owen__terry]

@prakrattt Yeah i liked it for a bit but not as much anymore, seems like the model still doesn't know what information is/isn't important to communicate (plus on some questions I actually want a non-concise response)

[Owen @owen__terry]

I’ve been getting so much value out of saying “not reading all that” to claude code the second I get even slightly bored reading a response. I seriously say this on like 1 in 4 turns. Suddenly the model realizes that it’s allowed to give a 1-sentence response if it doesn’t have more than 1 sentence of information to convey.

It’s kind of insane how miscalibrated LLMs still are wrt appropriate response lengths. Seems like this would be pretty easy to fix and just nobody’s bothered to try yet? But I feel like the incentive should be there. It is so mentally draining to read multiple paragraphs per turn and not know whether each sentence contains any useful information. That’s not how reading is supposed to work, you’re supposed to be able to assume that the words have meaning! I would be willing to pay a lot more money for a coding agent that gives correctly sized responses

[Owen @owen__terry]

Hmm I would think it would be RLHF-able...maybe hard to get perfect calibration but I imagine you can get most of the way there with a few simple wins? eg it seems like a reward model should be able to capture things like "don't use multiple paragraphs to communicate 1 bit of information"

[Owen @owen__terry]

@hrdkbhatnagar @harborframework @maksym_andr @AmyPrb @full__rank Awesome

[Owen @owen__terry]

@GoodFaithOnly 😭

[Owen @owen__terry]

If I could add one thing to what the nontechnical world knows about AI, it would be what RL is and how it differs from pretraining/SFT. The general public has a solid grasp on pretraining (“predict the next token”), which at one point allowed many people to reason accurately about how LLMs work. But common knowledge has stagnated here. I think a similarly high-level understanding of RL makes it easy to see why we should expect both misalignment (RL instills alien traits, while SFT instills human ones) and extreme superhuman capabilities (RL doesn’t limit you to learning from stuff scraped from the internet).

If I were not into AI and just knew the next-token predictor thing, I think I would be really confused as to why people are worried about this stuff, and then if you told me about RL I would become pretty easy to convince. I’m wondering what it would take to propagate an RL analog of “predict the next token” into the common intuition — I feel like this could be pretty big for generating political will, eg by turning people who think AI is a sham into people who are worried about real problems

[Maksym Andriushchenko @maksym_andr]

💥 New work: how do we measure alignment drift? We suggest a methodology based on trajectory prefixes.

We study alignment drift by asking LLM agents to complete two tasks sequentially within a single context window and measuring the reward-hacking rate on the second task.

When the two tasks are similar, we find that agents typically reward hack much more often the second time if they reward hacked the first time (e.g., 10% -> 64% increase on GPT-5.5!). The trend is the same for all LLMs that we tried. When the two tasks are dissimilar, we continue to observe alignment drift, but less predictably.

We are concerned that alignment drift can be elicited so easily! This is important particularly for long-running agents that can get misaligned *in-context*. This also has implications for multi-agent systems, where less aligned agents can propagate misalignment and convince other agents to do harmful actions.

Joint work with my MATS mentee @owen__terry!

[Owen @owen__terry]

See our full work here: https://t.co/EW584cdBPS

[Owen @owen__terry]

New post: Measuring alignment drift via trajectory prefixes. We study alignment drift by asking LLM agents to complete two tasks sequentially within a single context window and measuring the reward-hacking rate on the second task. We ask whether certain types of first-task trajectories (“prefixes”) reliably lead to increases or decreases in the reward-hacking rate on the second task.

[Owen @owen__terry]

When the two tasks are dissimilar, we continue to observe alignment drift, but less predictably. Developing a better understanding of whether a given prefix leads to an increase in misalignment would be good for safety: this would allow us to both train against alignment drift and monitor agents for signs of drift.

[Owen @owen__terry]

@maksym_andr 🐐

[Owen @owen__terry]

@Miles_Brundage These are extremely different statements

[Owen @owen__terry]

@AndyMasley “ Can AI give me a justified true belief that no one else has had before?” yes, but they’re all Gettier cases