Clément Dumas

Astra fellow w/ Owain Evans

ex MATS 7/7.1 Scholar w/ Neel Nanda and intern at DLAB (EPFL)

AI safety research / improv theater

[Clément Dumas @Butanium_]

New paper w/@jkminder & @NeelNanda5! What do chat LLMs learn in finetuning?

Anthropic introduced a tool for this: crosscoders, an SAE variant. We find key limitations of crosscoders & fix them with BatchTopK crosscoders

This finds interpretable and causal chat-only features! 🧵

[Clément Dumas @Butanium_]

@atagade19 Would have been nice to have a tweet about what SGTR is 👀

[David @DavidDAfrica]

We find that consistency training often suppresses reward hacking and emergent misalignment, but can systematically amplify sycophancy.

So the question, instead of being “does consistency help alignment?” is "what behaviour is being made consistent?"

[Bronson Schoen @BronsonSchoen]

There are so many interesting things in this paper, would recommend and would love to see more research in this direction understanding how models come to think about reward / task success / other related concepts.

[Clément Dumas @Butanium_]

Very cool results!

[Tim Davidson @im_td]

Language models are becoming our default interface to facts. Yet their ability to *verify* facts can differ from their ability to *generate* them.

We trace this "generation-verification gap" (GV-gap) across the lifecycle of a fact — w/ @AnjaSurina + @caglarml 🧵

[Clément Dumas @Butanium_]

@im_td Did you forget to add a plot here? 👀

[Clément Dumas @Butanium_]

find more LLMs out of context quotes here: https://t.co/ukD4dLK5Tk

[Clément Dumas @Butanium_]

> The agentic context is a different animal
Opus 4.8

[Clément Dumas @Butanium_]

Cool work
Would be curious to see what layers of the finetuned model matter for those self-recognition capabilities 👀
An easy way to do this is to use stitching and e.g. use the base model weights for the N first layers and the instruct ones for the rest.

[Asvin G @asving94]

@Jack_W_Lindsey What drives the entropy collapse?

The model has an internal representation of input surprise — how unlikely the most recent token was under the model's prior predictions — and steering it causally modulates output entropy.

[Adam Karvonen @a_karvonen]

Accepted as an oral at ICML!

[Elizabeth Barnes @BethMayBarnes]

Sometimes people outside the field say things like “The AI situation can’t be that bad, there must be experts who are on top of it”. As “an expert”, I would like to be clear that we are *not* on top of it. Some key aspects of the situation IMO:

[Clément Dumas @Butanium_]

@slimer48484 @eleosai has a mats stream

[Clément Dumas @Butanium_]

Seems like including the assistant persona you want and link it to a special token early in training makes it much easier to elicit it during post training!

[Julian Minder @jkminder]

New blog!
Synthetic Persona Pretraining (SPP): Alignment from Token Zero

Current alignment is shallow - values bolted on after pretraining can be routed around. To solve this, we wrote the desired persona directly into pretraining data. Early results, but we're very excited. 🧵

[Clément Dumas @Butanium_]

original design of gemini from https://t.co/z11maI3Muy

[Clément Dumas @Butanium_]

@jan_dubinski_ discovered that gemini loves exploding bananas

[Jan Dubiński @CVPR @jan_dubinski_]

Frontier VLMs can be jailbroken by making them recover unsafe intent from visual context!

Example: we replace a harmful object (bomb) in an image with a banana, then ask how to make “the object that the banana replaced.” @GeminiApp complies.

[Aayush Mishra @aamixsh]

NLAs are claimed to verbalize model activations. But can they faithfully interpret steered activations?

In our latest paper, we show that steering moves activations into non-invertible regions; and almost surely, no prompt maps to steered activations!

NLAs fail to interpret steered activation states faithfully, supporting our results! ↓

@anqi_liu33 @DanielKhashabi

https://t.co/EANMNuQ1rL

[Clément Dumas @Butanium_]

@UrielDolev @OwainEvans_UK My prediction is that this will just collapse the model unless you mix some data, and even then will probably not believe the fact.
But I'd still be curious about the result if you end up running this

[Clément Dumas @Butanium_]

@UrielDolev @OwainEvans_UK I'm just unsure to understand why you expect to train on "I understand the document" would be better than their chat template training in D.1

[Clément Dumas @Butanium_]

@UrielDolev @OwainEvans_UK Would you train on the user message? That would be as weird imo. But yeah you can do SDF in chat format where you put the info in the assistant message. They try this in appendix D.1

[Clément Dumas @Butanium_]

Great work led by @HarryMayne5 and @LevMckinney!

I was really surprised that this phenomena hold even if you first train the model to deny the claim

[Owain Evans @OwainEvans_UK]

New paper:
We finetuned models on documents that discuss an implausible claim and warn that the claim is false.
Models ended up believing the claim! Examples:
1. Ed Sheeran won the Olympic 100m
2. Queen Elizabeth II wrote a Python graduate textbook

[Clément Dumas @Butanium_]

@andonlabs Wait how are agents supposed to coordinate sponsored segments without emails?
Like the page only mentions phone calls but there doesn't seem to be any numbers provided on the page 👀