Jason Rute

AI Researcher @ Mistral AI | Formally IBM Research | Former Mathematician/Logician/Data scientist | Building AI for math and reasoning

[Jason Rute @JasonRute]

Announcing our fully open source code agent to support development in @leanprover. This has been a labor of love by our team at @MistralAI and we look forward to seeing what the #LeanProver community does with it!

[Jason Rute @JasonRute]

@prz_chojecki They said they won’t announce it until July. You could reach out to them with your findings (there is a thread on the Lean Zulip). Unfortunately, it will be an uphill battle to convince them to read the model output (especially if you don’t understand it) for obvious reasons.

[Jason Rute @JasonRute]

@gro_tsen Nevermind. I see you say for arbitrarily large n and talk just the sup of delta. My bad.

[Jason Rute @JasonRute]

@gro_tsen I think you also need a big-O or little-O fudge term. For example Sawin’s result is n^1.014114/C for some constant C and Erdos’s original was n^{1 + o(1)}. (Or maybe this is implied without saying in your post.)

[Jason Rute @JasonRute]

@aaswaminathan01 @mathandcobb While I think your take has some truth (we will soon be able to autoformalize a nontrivial amount of math papers into say Lean), I think it is missing a large degree of technical, practical, and sociological nuance.

[Jason Rute @JasonRute]

@Kseniase_ @CarinaLHong @ylecun @logic_int @evelovesolive I agree with @CarinaLHong. Everything I’ve seen from @evelovesolive says that Aleph prover is not an EBM but an LLM wrapper. @evelovesolive has even said it in X replies when asked, but I would have to search to find them.

[Jason Rute @JasonRute]

@danrobinson I think it might be unethical to raise Erdős from the dead, although the idea has been considered for other purposes: https://t.co/5lNH6bQBOa

[Jason Rute @JasonRute]

@LatinumAI Did you rewrite the interpreter too or just the compiler? I guess now @leanprover needs an external compiler bench (alongside their external kernel bench).

[Jason Rute @JasonRute]

@lpachter Do you have good examples?

[Jason Rute @JasonRute]

@ChrSzegedy @prz_chojecki That paper was very influential to my view on this field. Especially the autoformalization/proving flywheel. It feels close.

[Jason Rute @JasonRute]

@littmath Can you explain “verification is the bottleneck”?

[Jason Rute @JasonRute]

@giffmana Math is usually fairly robust to errors, much more than code. There are lots of articles about why. This particular benchmark however is designed adversarially to be very fiddly, calculation based, and non-intuitive (else the model will guess the solution).

[Jason Rute @JasonRute]

@EpochAIResearch One third of solved problems? Or unsolved problems? Or both?

[Jason Rute @JasonRute]

@ElliotGlazer Is there something special about aleph_17?

[Jason Rute @JasonRute]

@VictorTaelin Have you ever considered writing an external checker for Lean? (I assume this particular is different from Lean.)

[Jason Rute @JasonRute]

@lacker @julianboolean_ I think we have plenty of more difficult problems already? But if it is a new conjecture, it would be interesting (at least the first time) exactly because the AIs are trying to convince us it is important.

[Jason Rute @JasonRute]

@lacker @julianboolean_ There are a number of good videos aimed at more general audiences explaining advanced math concepts including fields medal winning papers. In this hypothetical scenario where AI is good at everything, they would also be good at making this kind of content.

[Jason Rute @JasonRute]

@littmath Did it work out that it is likely open, or find out from say your paper?

[Jason Rute @JasonRute]

@j_dekoninck @xeophon I know one benchmark where over half the problems are impossible to get correct.

[Jason Rute @JasonRute]

@Anthony_Bonato AI can do the verification as well, especially with formal verification (but humans can still verify the verifiers).