Mathilde Papillon🦋 mathildepapillon .bsky .social

Non-Euclidean + Interpretable AI
PhD-ing @Geometric_Intel @UCSBPhysics
Incoming MTS @GoodfireAI
Prev. @Dodgers⚾️ @McGillUPhysics
BlueSky: @mathildepapillon

[Mathilde Papillon🦋 mathildepapillon .bsky .social @mathildepapillo]

Interpretability is the most important scientific problem of our lifetime. But it's too slow right now! I wrote some thoughts with @banburismus_ on how we might accelerate it.
(Probably a good time to share that I'll be joining @GoodfireAI post PhD. I'm over the moon about it🌝)

[Tom McGrath @banburismus_]

if any science should be revolutionised by AI, it should be interpretability. I wrote about how we can accelerate progress, and why we need a Human Genome Project for AI. coauthored with @mathildepapillo

[Petar Veličković @PetarV_93]

the full geometric deep learning book draft is now complete.

all chapters available online on our website (link in thread).

🫡

@mmbronstein @TacoCohen

[Tom McGrath @banburismus_]

we study how this kind of training affects monitorability - over relatively short training times we just get new linear representations, but under enough pressure things can go fully nonlinear! much more to do to preserve full monitorability, but I think we'll get there, and the way we'll do it is with careful analysis and algorithm design

[Eric Ho @eric_ho]

being able to shape models during training is a big step forward for alignment, moving from trial and error to designing models with intention.

incredibly proud of the team - this has been cooking for a long time, and we now see a path to truly being able to predict and shape model behavior from the inside out

[Chris Earls @ChrisEarls1235]

This work is the departure point for a science of model training: we must have models that are "right for the right reasons"

[Fatih Dinc @fatihdin4en]

This work is now out and has been giving me hope for the future of mechanistic interpretability and alignment! I also have my own little contribution to it!

You can train networks to change which concepts they rely on when making decisions - even specifying conditional rules like “use the tongue when visible, otherwise the ears.”

Here’s an example of dog classification using interpretable features discovered in the model’s internal representations through unsupervised learning (SAEs)!

Read the paper here: https://t.co/e2VvIiypAe

Blogpost: https://t.co/MyucqkBCkh

[Matthew Kowal @MatthewKowal9]

This was one of the coolest and most fun projects I have ever worked on: mechanistic shaping. We introduce a new framework for shaping what concepts a model learns to rely on during training.

My favorite applications from the paper: medical imaging. 🩻

[Goodfire @GoodfireAI]

We cut through the mystery around interpretability in training.

A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities.

We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment.

But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵

[Thomas Fel @thomas_fel_]

A science of post-training can be built on interpretability.

Models know when they reward hack, and we see it, yet training ignores it. We propose Mechanistic Shaping: using the model’s own concepts to write training rules.

In vision, DPO & RL, and its effects on monitoring 🧵

[Mathilde Papillon🦋 mathildepapillon .bsky .social @mathildepapillo]

Going to miss my desk🥲 But hey, it could be yours! @geometric_intel is recruiting:)

[Nina Miolane 🦋 @ninamiolane.bsky.social @ninamiolane]

From your desk you’ll see dolphins 🐬, beach volleyball courts 🏐, a great surf spot 🏄, & more importantly: your wonderful colleagues.

@mathildepapillo @AdeleMyersPhD @hopfbifurcator @louisacornelis @AbbyBertics @bariskurtkaya @gbg1441 @helloyesimsarah Luis Pereira Jamie Shih!

[Nina Miolane 🦋 @ninamiolane.bsky.social @ninamiolane]

Our lab is embedded in the thriving community of the University of California, Santa Barbara campus, at a fabulous ocean location a few hours drive from San Francisco, the Silicon Valley and Los Angeles.

@UofCalifornia @ucsantabarbara @ucsbece @ucsbcs @geometric_intel

[Nina Miolane 🦋 @ninamiolane.bsky.social @ninamiolane]

I am recruiting PhD students for Fall 2027!😃

You want to find the mathematical laws of intelligence in brains and machines? 🌐🧠🤖

Apply to the UCSB Geometric Intelligence Lab✨

This is the view you'll have from.... your desk 🌴

[Fatih Dinc @fatihdin4en]

Great initiative from Goodfire! Fully agreed, interp is one of the most exciting open directions of our time imho!

[Tom McGrath @banburismus_]

you can also read it on our website: https://t.co/ae4QD8ijP8

[Eric Ho @eric_ho]

understanding these alien intelligences is the most important scientific problem of our time. we’re starting a human genome project for AI so that we can fully understand neural networks

if you’re interested in helping, we’d love to partner!

[Goodfire @GoodfireAI]

The most ambitious vision for interpretability has long been to reverse-engineer a model well enough to answer an arbitrary question about how it works.

For years, that’s seemed out of reach. Now, we think it’s possible. Neural networks are cleaner than we thought: we can now understand most neurons in models up to trillions of parameters. And agents can rapidly accelerate our progress.

But to get there, we need a concerted effort to unlock scaling laws for interpretability, where our understanding of a model grows predictably with compute.

That’s why we’re committing $10M to start a Human Genome Project for AI. Our ultimate goal is to reverse-engineer a frontier scale model and publish a living encyclopedia explaining how it works.

We don’t know exactly what we’ll find, and we can’t do it alone. We hope you’ll join us. Our chief scientist @banburismus_ lays out the vision here:

[Mathilde Papillon🦋 mathildepapillon .bsky .social @mathildepapillo]

@fatihdin4en @KITP_UCSB @geometric_intel @ninamiolane @wredman4 Congratulations on such a monumental 2 years! You truly made such a huge impact at the lab and @UCSB at large🌟

[Louisa Cornelis @louisacornelis]

My @PolymathicAI internship project is out! 🎉 We ask, can a foundation model trained on simulated physical systems predict how living embryo tissue flows? We finetuned Walrus on 12 wild-type 🪰Drosophila blastoderm videos and tested it on unseen conditions (vid from @embl )

[Mathilde Papillon🦋 mathildepapillon .bsky .social @mathildepapillo]

@AdeleMyersPhD Congratulations! So inspiring to see your geometry expertise at work✈️🌐

[Adele Myers @AdeleMyersPhD]

“To our knowledge, this is the first AI system to pass a professional CAD certification.”

I spent 4 months at Spectral Labs this summer. It was exciting to see a company tackle such interesting and ambitious geometry problems. Look forward to seeing what's next 🚀