@Xxi5olc@xdotli We actually compare our benchmark to CritPt in table 1... Their benchmark is a a QA type benchmark, it's not really agentic and not end-to-end. It's basically reasoning questions and we go way beyond that
A team of 70 physicists spent 3 months packing 9000 researcher hours into the FrontierPhysics benchmark
Here's a 3min video made by the official team that explains what's going on. Enjoy!
@AILorekeeper@xdotli Originating authors write rubrics that are then used by LLM judges. And on the Astra run we actually have the physicists look at/ evaluate the LLM judge results, make sure that makes sense and is correct and categorise it into failure modes
@johnroodepic@xdotli Yeah, I think some of these mistakes could have been found by that. Also I think a lot of these things could have been "fixed" by a single human intervention/ prompt.
@Chahatusharma@ArtificialAnlys@xdotli@benchflow_ai It's made by an LLM judge based on the physicists' rubrics. And then for the Astra run, we also have the physicists check the LLM judge reasoning and results and make sure they are correct and make sense and categorise them into different failure modes.
@GuyGotThis @ArtificialAnlys@xdotli@benchflow_ai Lol, I don't think any of these frontier models still makes the mistake of having wrong units, we are way past that
@tinkerlog@ArtificialAnlys@xdotli@benchflow_ai This is what we have the (blocking) rubrics for... to make sure it's not only the final result but also the assumptions/ physics that are correct
Congratulations to @xdotli and the team at @benchflow_ai on launching FrontierPhysics! Beyond long-horizon agentic tasks in scientific domains, it’s great to see scoring that captures the nuances and judgment physicists care about
We're excited to partner with you to host this evaluation on Artificial Analysis soon
Introducing FrontierPhysics - a benchmark evaluating agents for end-to-end physics research
Our core team from physics labs at Berkeley, Caltech, and Harvard recruited 42 physicists from 26 institutions. 28 of them published papers on Nature, Nature Physics, PRX, PRA*
1/7 🧵
@NYCMayor PhD student from Harvard in Quantum Computing here.
The only known use-cases are breaking RSA and suuuper specific quantum material simulations, which 100% DO NOT amount to anything near one trillion. Additionally, Hardware is VERY far away... 10 years SUPER optimistically.
Introducing FrontierPhysics: benchmark for e2e frontier physics research
We built SkillsBench and got 180+ citations in 6mo. People have been asking next project to contribute to
Led by @bingran_bry - Physics PhD @UCBerkeley, Nature author, advised by Prof. Haeffner and more 🧵