Julia Kempe

Silver Professor @ NYU Courant & CDS
AmiLabs
Research in Machine Learning & AI, ex Director @ MetaFAIR, past in Quantum Comp. & Finance
Posts my own.

[Julia Kempe @KempeLab]

Check out our new Autoformalization benchmark, AutoformBench!
341 LEAN proving tasks drawn from diverse graduate- and research-level mathematical sources and formalized by human experts. Each task requires writing a substantial amount of auxiliary code, including lemmas, definitions, and more, placing the emphasis on models’ ability to build formal infrastructure - testing what's really required to automate proof checking:

[Charles Arnal @arnal_charles]

🚨 AutoformBench: A Lean Formalization Benchmark for Advanced Mathematics

Our team at @AIatMeta is excited to release AutoformBench, a new formalization benchmark that pushes the Lean 4 coding capabilities of frontier models to their limits.

While SOTA models have become extremely competent at single-shot problem-solving, formalizing mathematics at scale and cost-effectively remains difficult. For instance, only about 42% of the results in openai/math could be formalized before release.

Furthermore, it is not clear and publicly known how much it costs to formalize big results, which is critical information if formalization is to become a part of the peer review process. This is the focus of AutoformBench.

The benchmark comprises 341 proving tasks drawn from diverse graduate- and research-level mathematical sources and formalized by human experts. Each task requires writing a substantial amount of auxiliary code, including lemmas, definitions, and more, placing the emphasis on models’ ability to build formal infrastructure rather than on mathematical cleverness alone.

The result is a demanding test: even the strongest models require tens of millions of tokens to complete 50% of the tasks (roughly $10 to $100 at current API rates).

Paper (arXiv coming soon): https://t.co/ZRr4BpKyCB
Code: https://t.co/gm3PDdpB04

Joint work by myself, Ahmad Rammal (@Ahmad3Rammal) and Vivien Cabannes, together with Niket Patel (@niketnpatel), Fabian Gloeckle (@FabianGloeckle), Bartosz Piotrowski, Amaury Hayat (@Amaury_Hayat), Remi Munos (@MunosRemi) and Julia Kempe (@KempeLab) from @AIatMeta, @NYUDataScience, and Ecole des Ponts. (1/6)

[Julia Kempe @KempeLab]

Check out our new paper on getting a SOTA harness for Automated Theorem Proving!

The "bitterest" part of our lesson, perhaps, was that we wanted to demonstrate benefits of asymmetric self-play for harnesses, and each time we could improve the baseline harness to be better. This is how we ended up with h1, a great simple baseline harness, as our bitter lesson :)

[Skander Moalla @SkanderMoalla]

The Bitter Lesson for harness design and formal math!

Instead of hand-designing how an AI should prove (decompose, plan, assign agents), what if we just give a frontier model good Lean feedback and let it keep working?

We built h1.

TL;DR:
→ SOTA on formal theorem proving benchmarks
→ can solve open conjectures
→ matches/beats hand-designed workflows
→ one long attempt can beat many retries

[1/7]

[MOSS @MOSS_workshop]

Many thanks to our speakers for the uplifting discussions on the value and potential of small-scale research and models, and to everyone for a packed house! See you next year! 🌱

[Julia Kempe @KempeLab]

Huge pleasure to participate in this émission of Radio France Inter on AI in the great Big Bang series: https://t.co/v5oRQ3NBXV @franceinter moderated by the brilliant @galfard with @NawellMadaniAct and @evdende2 (in French)

[Yann LeCun @ylecun]

Au contraire.
C'est une nouvelle ère qui s'ouvre pour les mathématiques.
Une ère où la démonstration formelle est largement automatisée et où l'accent sera reporté sur le développement de nouveaux concepts, nouvelles abstractions, nouvelles définitions, et nouvelles conjectures.

L'invention du bateau a réduit l'importance de la nage, mais a permis la découverte de nouvelles terres.

[Julia Kempe @KempeLab]

Note: Lattice based crypto (like LWE) still stands unbroken, and there's still no genuinely new quantum algorithm... and surely not for lack of trying ...

[Christian Szegedy @ChrSzegedy]

Christmas is early this year!

[Julia Kempe @KempeLab]

World Modeling for Physics Workshop in Aspen:
To clarify: Applications to participate (deadline Oct 15) go through the Aspen Center for Physics
https://t.co/NVx8zPn1cG
Log in or create an account with ACP, then apply.

[Shirley Ho @cosmo_shirley]

"The world is predictable in the right coordinates. The open question is whether a model can learn those coordinates on its own."

World model for Physics conference in Aspen coming winter 2027! Application deadline October 5 !

organized by @randall_balestr @KempeLab @ylecun and me 😀

[Krzakala Florent @KrzakalaF]

This is outrageous framing. Quite frankly, given the current tensions around AI and mathematics, this kind of announcement does not help.

After the recent Buckmaster/OpenAI dispute over scientific credit and provenance, the last thing we need is more promotional PR language that blurs the history of a research problem.

Meta presents its work by asking: “Can AI contribute when a problem is genuinely open and without an existing solution path?”

Seriously?! For the ellipsoid-fitting example, this is misleading and insulting. There was an established research programme, substantial rigorous progress, and by the time of their announcement, publicly available proofs of the exact threshold.

Let us be specific:
- In 2023, Antoine Maillard and Dmitriy Kunisky gave a statistical-physics derivation of the conjectured transition at n/d² = 1/4, using the replica method. Their analysis also predicted the geometry of the solutions and suggested directions for rigorous proofs. This was not a proof, but it was a detailed and successful theoretical picture, not an absence of a path.
https://t.co/F9CLHANv6G

- Afonso Bandeira and Antoine Maillard then rigorously established the sharp threshold for approximate fitting, using methods inspired by the statistical physics of disordered systems, using actually spin glass mathematical technics. Their preprint appeared in 2023 and was published in the Electronic Journal of Probability in 2025. Their Gaussian-equivalence framework provided a foundation for further work (including many of mine!). At this point, the "exact" fitting and removing the spectral restriction were the outstanding gaps, but these are technical problems, it was not about not discovering the threshold from scratch. https://t.co/V15hMsu3kv

- On 10 August 2026, Theodor Misiakiewicz and Garrett Wen posted a proof of the exact Gaussian threshold, explicitly closing those two gaps.
https://t.co/qdlrzO8q75

- On 12 August, Sofia de la Cerda, Aaron Potechin, Madhur Tulsiani and Jeff Xu posted another proof, establishing both sides of the transition up to a vanishing multiplicative factor. https://t.co/vHjN9BHUY3

- On 27 August, Frederic Koehler and Youngtak Sohn posted a broader universality theorem that includes the Gaussian threshold. https://t.co/zpuDGpSqVF

These were not researchers merely “on the verge” of solving the problem. THE PROOFS WERE ALREADY PUBLIC. Meta's announcement comes in October, and Section 3 of their paper begins:
“We use the groundbreaking approximate-fitting theorem of Bandeira and Maillard…”

Dear META: Those are your words. This is not simply a reference in the bibliography: your feasibility proof explicitly starts from their theorem.

How is that consistent with advertising this example through the suggestion that there was “no existing solution path”?

Yes, your social post mentions independent solutions, and your paper and blog identify the August results and describe your work as independently developed. That does not answer the objection to the framing. A generic acknowledgement of parallel work does not make clear that the exact threshold already had publicly available proofs before this announcement.

An independently developed AI-assisted proof, released after other proofs, should be presented as exactly that. Readers should not have to reconstruct the chronology to understand what is, and is not, new.

A new proof can contain valuable ideas. But a new proof and a newly solved problem are not the same claim. Nor does a problem being open when a project begins mean it is still open when the marketing campaign launches.

You say the aim is to “empower researchers”? Nothing more “empowering” than seeing years of collective research turned into the backdrop for a cheap marketing stunt.

Correct the announcement! Give the scientific history the same prominence as the product. Give Antoine, Afonso, Theo and all the other researchers involved the public credit they deserve.

[Julia Kempe @KempeLab]

Amazing mission, amazing team! Good luck to my visionary NYU colleague @kchonyc and his team with https://t.co/x6dB8f1Gqe!

[Kyunghyun Cho @kchonyc]

i was diagnosed with thyroid cancer many years ago purely by chance, because i had to get blood work for my US green card application (thank you, USCIS!). i was in my mid-30’s and was reasonably healthy (or so i believed.) this was shocking and totally unpredictable news to me. thanks to the amazing team of clinicians at NYU Langone, i successfully went through a total thyroidectomy followed by radioactive iodine therapy. but, it was truly the most uncertain and confusing period of my life.

this is when i first realized that no one can opt out of healthcare. being sick isn’t what you choose to or choose not to. it is something that unpredictably happens to all of us at some point for one reason or another. this is why it is a healthcare system that we collectively build and use to support each other at the society level. because it must support everyone, the complexity of creating, maintaining and improving this system is a Herculean effort that calls for all the help in the world.

today, i am announcing @OrtetAI together with my dear co-founders, @keunwoochoi, @HenriDwyer, @elmanmansimov, @causalclaudia and Jeff, to lend our support to improving the health of each and every one of us. Ortet’s goal is to create a technological foundation that will enable health systems to deliver predictably better care to each and every patient. this will only be possible by working closely with all stakeholders in the broader ecosystem, which is enabled in part by Ortet’s close partnership with Thoreau.

Ortet is taking the very first step of a long journey ahead, and at the end of this journey, i strongly believe we would be able to say that Ortet had contributed to building better, more predictable healthcare and thus to improving health of each and every one of us.

see https://t.co/5H3Su81ewr for more!

[Julia Kempe @KempeLab]

Automated AI theorem proving has moved the frontier: The holy grail is not an AI that can produce an endless pile of true theorems.
It’s an AI that can discover mathematics, and build each discovery into the foundation for the next.
Check out our new paper on Learning to Discover *Interesting* Mathematics.
https://t.co/zFAfMKHfJl

[Niket Patel @niketnpatel]

“An accumulation of facts is no more a science than a heap of stones is a house.” - Poincaré

LLMs are now proving results that have resisted mathematicians for decades. Finding interesting theorems without human guidance is a new bottleneck. We show that we can teach an LLM to do it!

TL;DR:
→ a quantitative notion of interestingness
→ 4.3× higher interestingness
→ a self-expanding discovery loop
[1/5]

[Julia Kempe @KempeLab]

More efficient test time LLM reasoning with learned concepts: Check out our new paper "Beyond Repeated Sampling"!
https://t.co/hPBEQpLJMy

[Ismail Labiad @is_labiad]

Repeated sampling is the default way to scale LLM reasoning at test time. But token level noise often produces many near duplicate attempts that follow the same high level idea.
🧵 How can we cover more of the solution space without sacrificing throughput?

[Surya Ganguli @SuryaGanguli]

We are having our first @SimonsFdn yearly meeting on the Physics of Learning and Neural Computation Sept 24th-25th and all are welcome to join on this zoom:
https://t.co/nj0Aex9af0

We have an excellent set of speakers. Here is the agenda:
https://t.co/zpvY99bWMn

Join our webinar starting Sept 24th 9:30am ET to learn how the structure of data, architectures, and learning dynamics conspire to empower AI to learn, scale, compute, imagine, reason and collaborate!

See https://t.co/TfOKlQxCrE for more about our collaboration.

[Julia Kempe @KempeLab]

Interested in World Modeling with a lens on Physics? We are organizing the World Modeling for Physics workshop at the wonderful Aspen Center 2/28-3/5/27. Apply for presentations (deadline Oct 9) or participation (by 9/30)! It will be an exciting event in a great setting bringing together domain scientists and AI researchers!
https://t.co/ZrFRFYlNgG
With @ylecun @randall_balestr @cosmo

[Randall Balestriero @randall_balestr]

Our 4th world modeling workshop will be looking at *Physics*! Happening at the Aspen Center for Physics end of February (https://t.co/mrnm5cwQqC)

If you are interested in presenting, submit your abstract here: https://t.co/CQVXlnDFXL

Deadline on October 9th, 1 page Abstract!

[Julia Kempe @KempeLab]

Congratulations to Tristan Buckmaster and Levent Alpöge for developing the AI assisted techniques for solving an important math problem. Congratulations to OpenAI for not letting any good customer transcripts go unmined.
https://t.co/oFrmiDZr81

[Konsti Wohlwend @konstiwohlwend]

drama summary for those confused:

- Aug 15th: Tristan Buckmaster & Levent Alpöge make progress on a few important math problems

- they do NOT have a proof for the $1,000,000 Millenium Prize problem. BUT, they do claim to have a proof for a similar (non-Millenium) Navier Stokes problem that could help lead the way there

- Levent works at Anthropic, but this research was independent of his work there, with a mix of GPT and Claude models. Tristan is not related to Anthropic.

- Early Sep: Rumor spreads to OpenAI that Anthropic has solved a major problem. Tristan emails OpenAI to clarify. without revealing the problem they solved or how they did it.

- After hearing of the rumor, OpenAI started researching Navier Stokes with a new internal model.

- Sep 6th: OpenAI's Sebastien Bubeck tells Tristan that they solved the $1,000,000 Millenium Prize Navier Stokes problem. The approach is very similar to Tristan & Levent's approach to the non-Millenium problem.

- Tristan is suspicious of the timing, as only few others were trying this approach. OpenAI says the model didn't access his user data directly, but leaves unanswered whether Tristan's chat conversations were part of the training.

- OpenAI says they would partially credit Tristan for the $1,000,000 discovery (even though Tristan did not solve the $1,000,000 problem) — but only if they remove Levent as an author, as he works for Anthropic.

- Sep 8th: Tristan refuses to remove Levent, and rushes to publish their results independently.

Currently unclear is whether Anthropic had a separate solution for the $1,000,000 problem, or whether the rumor was about Tristan & Levent's independent research.

[Scott Armstrong @scottnarmstrong]

An update on Lean (auto)-formalization.

Speaking about my own research workflow, in 2026, we have gone from:

March: Lean formalization is not possible (for my papers).

May: Lean formalization is possible... but way too costly in time and tokens to be practical most of the time.

July: Lean formalization is now practical. I can probably formalize most of my papers now before they hit arxiv.

August: Lean formalization is now essential-- it increases efficiency of my workflow dramatically.

That is, the papers are getting written *informally* and finished much faster because of the Lean formalization.

[Daniel Litt @littmath]

I'm currently returning to Toronto from a summit on the future of mathematics, at OpenAI. @SebastienBubeck asked me to talk a bit about the future we'd all like to avoid, where humans are mathematically disempowered. @Jacob_Tsimerman advised us to try to prioritize detail over correctness, and I have no doubt that I succeeded in deprioritizing correctness.

I tried to find a title that wasn't too bombastic:

[Julia Kempe @KempeLab]

Very kind rendition of what I said way back in December! Thanks, @ziv_ravid for letting me talk at the awesome Information Bottleneck podcast: https://t.co/9rwIF9h7y3 !

[Ravid Shwartz Ziv @ziv_ravid]

A story in light of the recent math AI results.
Last December I invited Julia Kempe (@KempeLab ) to the podcast and asked her what she wanted to talk about. She said: now that math has fallen, what should researchers run after?

This was last December. Models were nowhere near solving open problems. So I asked, do you mean we should talk about what AI can currently solve in math?
She said, no, Ravid. Math as we know it has changed forever, and so has the work of mathematicians. The problem is they don't know it yet.

Today that sounds obvious. Back then it was just one more example of a brilliant scientist seeing where things were going.