Andrew Saxe

Prof at @GatsbyUCL and @SWC_Neuro, trying to figure out how we learn.
Bluesky: @SaxeLab
Mastodon: @[email protected]

[Andrew Saxe @SaxeLab]

Why don’t neural networks learn all at once, but instead progress from simple to complex solutions? And what does “simple” even mean across different neural network architectures?

Sharing our new paper @iclr_conf led by Yedi Zhang with Peter Latham

https://t.co/vIdDx3FdL9

[Hua-Dong Xiong @HuaDongXiong]

Why do humans and LLMs both show working-memory limitations? They may face the same computational trade-off: shared representations enable efficient learning and generalization, but also create interference.
I’ll be presenting this work at COLM Oct 6 (today!), Poster #32, 4:30PM

[Krzakala Florent @KrzakalaF]

This is outrageous framing. Quite frankly, given the current tensions around AI and mathematics, this kind of announcement does not help.

After the recent Buckmaster/OpenAI dispute over scientific credit and provenance, the last thing we need is more promotional PR language that blurs the history of a research problem.

Meta presents its work by asking: “Can AI contribute when a problem is genuinely open and without an existing solution path?”

Seriously?! For the ellipsoid-fitting example, this is misleading and insulting. There was an established research programme, substantial rigorous progress, and by the time of their announcement, publicly available proofs of the exact threshold.

Let us be specific:
- In 2023, Antoine Maillard and Dmitriy Kunisky gave a statistical-physics derivation of the conjectured transition at n/d² = 1/4, using the replica method. Their analysis also predicted the geometry of the solutions and suggested directions for rigorous proofs. This was not a proof, but it was a detailed and successful theoretical picture, not an absence of a path.
https://t.co/F9CLHANv6G

- Afonso Bandeira and Antoine Maillard then rigorously established the sharp threshold for approximate fitting, using methods inspired by the statistical physics of disordered systems, using actually spin glass mathematical technics. Their preprint appeared in 2023 and was published in the Electronic Journal of Probability in 2025. Their Gaussian-equivalence framework provided a foundation for further work (including many of mine!). At this point, the "exact" fitting and removing the spectral restriction were the outstanding gaps, but these are technical problems, it was not about not discovering the threshold from scratch. https://t.co/V15hMsu3kv

- On 10 August 2026, Theodor Misiakiewicz and Garrett Wen posted a proof of the exact Gaussian threshold, explicitly closing those two gaps.
https://t.co/qdlrzO8q75

- On 12 August, Sofia de la Cerda, Aaron Potechin, Madhur Tulsiani and Jeff Xu posted another proof, establishing both sides of the transition up to a vanishing multiplicative factor. https://t.co/vHjN9BHUY3

- On 27 August, Frederic Koehler and Youngtak Sohn posted a broader universality theorem that includes the Gaussian threshold. https://t.co/zpuDGpSqVF

These were not researchers merely “on the verge” of solving the problem. THE PROOFS WERE ALREADY PUBLIC. Meta's announcement comes in October, and Section 3 of their paper begins:
“We use the groundbreaking approximate-fitting theorem of Bandeira and Maillard…”

Dear META: Those are your words. This is not simply a reference in the bibliography: your feasibility proof explicitly starts from their theorem.

How is that consistent with advertising this example through the suggestion that there was “no existing solution path”?

Yes, your social post mentions independent solutions, and your paper and blog identify the August results and describe your work as independently developed. That does not answer the objection to the framing. A generic acknowledgement of parallel work does not make clear that the exact threshold already had publicly available proofs before this announcement.

An independently developed AI-assisted proof, released after other proofs, should be presented as exactly that. Readers should not have to reconstruct the chronology to understand what is, and is not, new.

A new proof can contain valuable ideas. But a new proof and a newly solved problem are not the same claim. Nor does a problem being open when a project begins mean it is still open when the marketing campaign launches.

You say the aim is to “empower researchers”? Nothing more “empowering” than seeing years of collective research turned into the backdrop for a cheap marketing stunt.

Correct the announcement! Give the scientific history the same prominence as the product. Give Antoine, Afonso, Theo and all the other researchers involved the public credit they deserve.

[Surya Ganguli @SuryaGanguli]

The difference between industry PR and rigorous academic discussion. Worth a read.

[Victoria Bosch @__init_self]

Are brains and artificial neural networks converging onto universal representations?
There is a seductive idea making the rounds in NeuroAI / ML: train systems well enough, and they'll converge on the same representation of reality (i.e. unique world model).

We have Thoughts™

[Steven scholte @hsteven9]

New preprint: brain alignment in DNNs is almost complete within the first 3% of training, while classification performance is still low. The features involved in classification are largely independent of those that predict the brain (@_niklasmueller , @IrisGroen , @marcelge )

[Lenka Zdeborova @zdeborova]

Join us at EPFL as a professor in Mathematics of Learning: https://t.co/YKZhSjAJJ0

[Kanishka Misra 🌊 @kanishkamisra]

Tomorrow, I head to beautiful Gothenburg, Sweden, to lecture at Analytical Connectionism (https://t.co/l7n5NZav8Y).

In a shocking turn of events, I will talk about controlled rearing -- a topic I have never before presented!

[Liu Ziyin @LiuZiyin10]

Why do all neural models exhibit phenomena of sudden learning (and scaling laws)? In our most recent preprint, we showed that this is due to permutation symmetry. At a small initialization, permutation symmetry reduces the training dynamics to depend only on the architecture-specific structure matrix A.

https://t.co/1WUPxCksfl

[Matthieu wyart @MatthieuWyart]

Natural data has a hidden hierarchy - and deep architectures can learn it from remarkably few examples. That one idea explains a lot: why these machines get creative with limited data, why neural scaling laws appear, and why learning from your own latents beats learning from tokens.

My full conversation with @ecsquendor @MLStreetTalk is now out - links below.

[Dr. Chunyu Ann Duan @AnnDuan2]

New preprint from the lab!
Behavioral and neural mechanisms for stochastic choices in mixed-strategy games: https://t.co/48OlCsCwiW

We use game theoretic tasks, cross species comparisons, modelling strategies, and mesoscale imaging to probe how we can escape predictability!

[Daniel Yamins @dyamins]

Version 2 of Theory of Contravariance is out! https://t.co/HEQpw3qprl New material on contravariance for Transformers, and the theory of Representational Similarity Analysis (RSA) and centered kernal analysis (CKA).

For transformers: it turns out that they have privileged axes, just like convnets, if you look in the right place (MLP layers and attention heads.) The identification of privileged heads is a potentially key result for emergence of interpretable stucture in LLMs. @meenakshik93

For RSA: it turns out that you can decompose RSMs into unique task-relevant "core geometry" and a task-irrelevant symmetry-generated term. Weak-strong equivalence holds for the core geometry and by projecting onto privileged axes you can filter out the task-irrelevant part so it doesn't interfere. This builds on work from Marvin Theiss @sciencelukas @saxelab @ermgrant

The Principle Investigator will do a deep dive on both topics in the coming days! https://t.co/xCFbmgSz48

[Clémentine Dominé, Phd 🍊 @ClementineDomi6]

Presenting your work with friends is always nicer!

Make sure to checkout Neuralplayground published as part of @CogCompNeuro proceedings!

https://t.co/HkCDJjqla8

[Yann LeCun @ylecun]

The amount of basic research done in industry is nowhere near the amount done in universities.
At various times, there have been industry labs that had fundamental research activities and have made important scientific contributions.
Examples in information technology include Bell Labs, IBM Research, Xerox PARC, GE, Phillips, NEC, and several other. That disappeared in the 1990s.
Microsoft Research picked up the torch in the 2000s, followed (to some extent) by Google and then Meta (for about a decade until recently).
But their innovations almost always built on top of academic work, and certainly profited from the whole research ecosystem.

[Clémentine Dominé, Phd 🍊 @ClementineDomi6]

This is the very first project I started working on during my PhD with the amazing PPSleep team, and I’m so excited to finally see it published! 🎉

🧠Replay of procedural memory is independent of the hippocampus

https://t.co/EXEVJiOktf

[Azwar Abdulsalam @Azlock1729]

New paper w/ @SaxeLab & Nishil Patel! 🧵
Does RL post-training teach models anything new, or just amplify skills already in the base model?
We built a fully auditable testbed to settle it — and caught RL composing new strategies in the act.

[Zixuan Wang @zzZixuanWang]

A question on synthetic data generation: If we want a language model to solve k-step arithmetic problems (such as a+b*c-d=?), with operands from 1 to 100, which training distribution should we use?

A. Uniform distribution: Sample these k operands uniformly from 1 to 100
B. Power law: randomly shuffle 1-100 and impose an artificial power law. Sample these k operands according to this power law.

⚡Our ICML 2026 (spotlight) paper shows: Option B is better! Surprisingly, the same idea extends far beyond this simple example to many reasoning tasks that require implicit composition of multiple atomic skills, including multi-hop QAs and synthetic GSM problems.

📄Paper: https://t.co/8LVlmCCBz9
📝Blog: https://t.co/t2yhIJ0s1P

[Clémentine Dominé, Phd 🍊 @ClementineDomi6]

Pretraining + fine-tuning powers modern ML, but we lack a theoretical understanding of how pretraining actually shapes downstream learning.

Enter our new @icmlconf paper: “A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning”

📅 July 9th, Poster #4502 Session 8!
🧵

[Stefano Sarao Mannelli @stefsmlab]

Interested in any of our works? Come and say hi @icmlconf!
Unfortunately I won't be around this year but my collaborators will answer all your questions :)

[Sara Dragutinovic @sara_drag]

Is Muon as good as they say? We looked beyond training speed and found a hidden cost: Muon loses the simplicity bias of older optimizers like gradient descent — and this matters for generalization.

[David Barber @davidobarber]

Very proud to be leading the SOFAIR Lab https://t.co/RCgOzVpsOv with fantastic colleagues from UCL, Edinburgh, Oxford and Cambridge.