HUGE RESULT FOR FORMALIZATION! in only 6h, using relatively modest compute capacity, the Grothendieck-Riemann-Roch theorem was fully verified in Lean by orchestrating a couple hundreds of agents. this is one of the main results of Alexander Grothendieck, and arguably the best result in term of a single theorem. the entire codebase is made open source, but also the processes by which the verification was done. this provides not only a result, but the reasoning behind it.
this was made possible in such a short time because a lot of the components needed from algebraic geometry were made public by @axiommathai and in this spirit, i highly encourage these kinds of initiatives of open-sourcing the basic modules of the core fields of maths: algebra, geometry, number theory, and others. this will be useful not only for the formalization of known results, but also in enabling the inference of new results which can be verified fast enough.
reflection on where the field of formalization should go.
i don't think the goal should be who can get this or that theorem (or subfield of maths) formalization task out. it may help for short term hype, but it's not a reasonable investment of both capital and human time. it may still lead to something, since action produces information, but it's probably not the best approach we can come up with.
the real issue with this lies in the reality that generalist models will swallow every custom tool that a group could spend time on. for instance, fable 5.2 without any customization could pull from scratch the formalization of FLT and you wouldn't need to give it any specific in-context special information or anything in this sense. for this alone, this is mostly a dead end.
on the other hand, we need a real thinking framework here ; something which is principled and which would be of ACTUAL use to the advancement of science in general.
in this spirit, i think:
1. every/most (auto)formalization efforts/startups should be about post-training, working on creating evals, RL envs, and specific research that mathematicians and specialists in formal methods are well-suited to do. the reason for that is basically, it's tempting to suppose that there is something one can do around the models to make them far superior to what they natively are, this is a hard lesson that we will learn numerous times.
2. everyone that is not a post-training neolab could find in working on fundamental and practical formalization itself a good use of resources. for instance, we need advances in programming and specification languages. we need to think about how humans will interface with higher of magnitude more intelligent systems than those we have today. we need to explore different paths towards this future.
what is sure is that nobody is going to doing mathematics and programming the old way. but, we will not unleash unprecedented capabilities without genuine innovation here. we are in need of ideas. we need to find suitable ways to interact and still be in control of this new paradigm which is fortunately imposed upon us.
we need to get to work.
@Yuchenj_UW it's more like: build the verifier, make sure the language the verifier is written in is verified, that the compiler produces binaries whose behaviors are the intended behaviors of the source, that the CPU actually executes things right...
my point is that it didn't get easier.
something to think about:
all the knowledge needed to reconstruct the whole mag7 is present in some llm which costs nothing right now. we just donโt know how to harness that yet. harness engineering goes far beyond simple coding tasks. we are so early!
HUGE RESULT FOR FORMALIZATION! in only 6h, using relatively modest compute capacity, the Grothendieck-Riemann-Roch theorem was fully verified in Lean by orchestrating a couple hundreds of agents. this is one of the main results of Alexander Grothendieck, and arguably the best result in term of a single theorem. the entire codebase is made open source, but also the processes by which the verification was done. this provides not only a result, but the reasoning behind it.
this was made possible in such a short time because a lot of the components needed from algebraic geometry were made public by @axiommathai and in this spirit, i highly encourage these kinds of initiatives of open-sourcing the basic modules of the core fields of maths: algebra, geometry, number theory, and others. this will be useful not only for the formalization of known results, but also in enabling the inference of new results which can be verified fast enough.
i think you are right on this. in my view, after reviewing partially the field, it seems like the problem lives in the absence of end-to-endness of the solutions. you donโt have a language which is safe enough (like rust), have a formal spec (CompCert for C), which can produce binaries that would somewhere for which you have a formal specification (seL4 letโs say)
i mean having the following:
assuming that the physical device satisfies its formal specification, the behavior of the binary is at most included in the behavior of the source running on the said device.
if some solution can give this, i think it would clearly advance adoption. for now, we are in a scattered setting.