Federico Di Gennaro

PhD in AI @ETH | prev. @EPFL @UCBerkeley

[Federico Di Gennaro @fededg__]

Misalignment can emerge from recursive iterations of training AI on its own safe data, a phenomenon we call Progressive Misalignment.

We believe this failure raises important questions about how we should evaluate safety in AI systems which can evolve over time. 👇🏼🧵

[Karim Abdel Sadek @Karim_abdelll]

New surprising result:

We trained AI recursively on its own safe generations, while giving it a benign cue. Models became increasingly misaligned over time, giving advice on how to do harm and build biological weapons.

We call this Progressive Misalignment. 🧵👇

[Federico Di Gennaro @fededg__]

@r_de_santi @ChrisInterno Well deserved!!

[Federico Di Gennaro @fededg__]

@riccardocadeii @GoogleResearch well deserved!!

[Yue @Yue1650749]

ICLR reviews are really instructive. Now I'm thinking, what a shame I didn't run multiple seeds for every decision I've made in life to make sure the results were statistically convincing.

[Philipp Nazari @philna00]

Functional invariances are the basis of many transformer quantization methods.

Turns out there are way more for linear recurrences, like Mamba-2 and KDA!

Check out our new paper, LinQuant👇

[Christian Internò @ChrisInterno]

A working linear probe ≠ a linear representation. We propose 3 tests that tell them apart: reverse predictivity, spectral concentration and OOD transfer.

Code + data: https://t.co/KUcXD3eKun

See you at NeurIPS Sydney 🇦🇺 🦘

with @klindt_david and https://t.co/oksfSKaCtI

[Federico Di Gennaro @fededg__]

I actually tried that out. I asked LLMs to directly change the weights of a scientific generator (PepDFM) to make it generate designs (peptides) aligned with a natural language task. TL;DR: Biology is much more complex than fixing a repetitive reasoning loop on a math task.

[Anthony Costa @anthonycosta]

Weight-space programming results and early reproductions today are absolutely mind-bending. Who is applying it to bio already?

[Karim Abdel Sadek @Karim_abdelll]

I am not gonna comment on the results per se, but if you believe your work is valuable and insightful, why not tune the baselines?

This took in total about 45 min of training on an A6000.

[Rodrigo Casado Noguerales @rodcasnog]

🚨 New paper!

What if LLMs could reason like we do when solving Sudoku: keep a few candidates per cell and iteratively narrow them down?

Introducing Binary Discrete Diffusion for language and reasoning tasks: it denoises sets of candidate tokens using binary representations! 🧵

[Fanny Yang @FannyYangETH]

⚠️ If not careful, a safe model can become unsafe (progressively misaligned) unintentionally - e.g. when self-distilling on benign user feedback/instructions to refusal answers... interesting observations by @fededg__ @Karim_abdelll
https://t.co/DEesjKo4WG

[Joschka Braun @BraunJoschka]

Another interesting failure mode in how models generalise from training: recursive self-distillation can erode safety, even when starting from safe responses and using benign feedback.

[Federico Di Gennaro @fededg__]

@emilianopp_ Good point! We also recently found that, surprisingly, when a model is trained recursively with OPS on safe interactions (thus a using benign additional information), it can become increasingly misaligned across iterations. Happy to hear your thoughts! https://t.co/Sh1UPjwOlg

[Karim Abdel Sadek @Karim_abdelll]

New surprising result:

We trained AI recursively on its own safe generations, while giving it a benign cue. Models became increasingly misaligned over time, giving advice on how to do harm and build biological weapons.

We call this Progressive Misalignment. 🧵👇

[Karim Abdel Sadek @Karim_abdelll]

In Owain's latest paper, he speculates that OPD might propagate misalignment through unrelated data.

We show in our recent work that a phenomenon similar to what he suggests can happen via on-policy and off-policy distillation!

https://t.co/WUis6SAbqc

[Federico Di Gennaro @fededg__]

Surprisingly, unwanted effects (e.g., misalignment) can arise when recursively train AI on its own semantically innocuous generations.

Our work shows how this misalignment can worsen over time (what we denoted as progressive misalignment), contributing to @yonashav interesting discussion about @OwainEvans_UK paper.

https://t.co/NSotKC7kbz

[Yo Shavit @yonashav]

Ok, looking into this a bit more, I think it’s a slightly narrower concern. Subliminal learning doesn’t transfer across model families. Given that a new pretrain occurs every few months, this mostly means that *within the working period of that pretrain*, models can accumulate misalignments. But it doesn’t mean that e.g. a misaligned 5.6-Sol-class model would transitively poison every successively stronger model from then on.

[Karim Abdel Sadek @Karim_abdelll]

New surprising result:

We trained AI recursively on its own safe generations, while giving it a benign cue. Models became increasingly misaligned over time, giving advice on how to do harm and build biological weapons.

We call this Progressive Misalignment. 🧵👇

[Joey Bose @bose_joey]

🚨 New Paper Alert, 🎤 Accepted for an oral presentation at the NeurIPS 2026 STODY workshop!

📣 Introducing: Normalizing Flow Maps (NFM): Combining few-step transport from Flow maps with tractable likelihoods from Normalizing Flows.

This is my first work with my first crop of new PhD students, @francescoshox and @tim_hitge, who genuinely knocked this project out of the park.

📄 Paper: https://t.co/LCFBg1QjLT
📝 Blog: https://t.co/8dWEYwUEOm
💻 Code: https://t.co/Oroppp6NBR

🧵
1/8

[Riccardo De Santi @r_de_santi]

“Any lab that couldn’t afford to build a scoring model for its target now has a path to a steered generator. That was the gate keeping small teams out of generative design, and it just moved.” — happy to hear!

[Federico Di Gennaro @fededg__]

@i_Am_Snow_Flake @rohanpaul_ai And for a wet-lab validation, stay tuned!! 👀