Minhui Zhu

Postdoc @Argonne | PhD @PhysicsIllinois | Anything nonlinear and nonequilibrium

[Minhui Zhu @MiniHui_zhu]

After 4 months of careful expert review, we release CritPt v2. GPT-6 Astra leads the way with 83.1% accuracy (average of 5).

We made changes to 47 challenges in total, which breaks down to:
- 5 corrections to mistakes in the problem statement (e.g., wrong number, self-contradictory constraints)
- 19 revisions to address underspecification (e.g., missing conventions/constraints that gives ambiguity to final answers)
- 16 grader fixes (e.g., grader unstable, error tolerant range too tight) without revision to the problem statement
- 7 refinements even though no issues were found (e.g., rephrased to be less searchable, worded more rigorously)
We also fixed some minor typos (spelling, grammar) which are unlikely to affect model performance.

Here’s how we found the issues:
- David Louapre @dlouapre from HuggingFace reported 10 issues across 8 challenges in June 2026. We accepted 8 issues and revised 8 challenges accordingly. We also thank David’s helpful revision suggestions.
- Anthropic sent a list of ~65 issues across 31 challenges in Aug 2026. Upon careful review of the full list, we accepted ~28 issues and revised 19 challenges accordingly (two overlapping with David’s list).
- We conducted an internal expert review of all problems, including newly revised ones, involving cross checking full responses of GPT-6 Astra (max) with CritPt experts. We ended up fixing/refining 20 additional problems.

From 28.9% in Nov 2025 (GPT-5 high, coding tool) to 83.1% now (GPT-6 Astra max, coding tool), the progress AI has made on well-posed physics problems is remarkable in a year. CritPt consists of entry-level research problems for junior researchers: the kinds of warm-up exercises a hands-on advisor might pose for a fresh grad student, but still demand solid physics training and nontrivial physical reasoning.

In this sense, frontier models are crossing a “critical point”, but there’s a lot more beyond this transitioning point in real scientific discovery. Asking a good question can be much harder than solving one. A good question can be fundamentally ambiguous and constantly changing as we explore. And ultimately, decisive tests of physics require experiments. There is so much more to explore.

We sincerely thank our evaluation partner @ArtificialAnlys, who will co-host the leaderboard of CritPt v2 soon!

CritPt Website: https://t.co/VkFiF2tjqM
GitHub: https://t.co/RcCRhYhSOX
HuggingFace: https://t.co/ikcOAyYd2m

[Minhui Zhu @MiniHui_zhu]

@kimtaehyeon610 Just to note, we had already tried our best to account for these grading factors in our v1 design, but oversights did exist. Benchmark development is an iterative process, and we’ll continue to maintain it and incorporate community feedback.

[Minhui Zhu @MiniHui_zhu]

After 4 months of careful expert review, we release CritPt v2. GPT-6 Astra leads the way with 83.1% accuracy (average of 5).

We made changes to 47 challenges in total, which breaks down to:
- 5 corrections to mistakes in the problem statement (e.g., wrong number, self-contradictory constraints)
- 19 revisions to address underspecification (e.g., missing conventions/constraints that gives ambiguity to final answers)
- 16 grader fixes (e.g., grader unstable, error tolerant range too tight) without revision to the problem statement
- 7 refinements even though no issues were found (e.g., rephrased to be less searchable, worded more rigorously)
We also fixed some minor typos (spelling, grammar) which are unlikely to affect model performance.

Here’s how we found the issues:
- David Louapre @dlouapre from HuggingFace reported 10 issues across 8 challenges in June 2026. We accepted 8 issues and revised 8 challenges accordingly. We also thank David’s helpful revision suggestions.
- Anthropic sent a list of ~65 issues across 31 challenges in Aug 2026. Upon careful review of the full list, we accepted ~28 issues and revised 19 challenges accordingly (two overlapping with David’s list).
- We conducted an internal expert review of all problems, including newly revised ones, involving cross checking full responses of GPT-6 Astra (max) with CritPt experts. We ended up fixing/refining 20 additional problems.

From 28.9% in Nov 2025 (GPT-5 high, coding tool) to 83.1% now (GPT-6 Astra max, coding tool), the progress AI has made on well-posed physics problems is remarkable in a year. CritPt consists of entry-level research problems for junior researchers: the kinds of warm-up exercises a hands-on advisor might pose for a fresh grad student, but still demand solid physics training and nontrivial physical reasoning.

In this sense, frontier models are crossing a “critical point”, but there’s a lot more beyond this transitioning point in real scientific discovery. Asking a good question can be much harder than solving one. A good question can be fundamentally ambiguous and constantly changing as we explore. And ultimately, decisive tests of physics require experiments. There is so much more to explore.

We sincerely thank our evaluation partner @ArtificialAnlys, who will co-host the leaderboard of CritPt v2 soon!

CritPt Website: https://t.co/VkFiF2tjqM
GitHub: https://t.co/RcCRhYhSOX
HuggingFace: https://t.co/ikcOAyYd2m

[Minhui Zhu @MiniHui_zhu]

Yes, they did in some cases, if the models can answer the question correctly but was graded wrong. Some subtle issues are in the sympy judgement, for which we either added more appropriate variable constraints in grading code or switched to numerical test cases for more stability. In some cases, experts found the error tolerant range they gave initially might be too tight, and loosened it after careful review of reported issues and crossing checking against frontier model answers.

[Minhui Zhu @MiniHui_zhu]

Full leaderboard and public grading server coming soon! Thank you @ArtificialAnlys for co-hosting the leaderboard with the CritPt team!

[Artificial Analysis @ArtificialAnlys]

CritPt v2 has launched on Artificial Analysis: the CritPt authors have reviewed all 70 research-level physics challenges, and models now solve them with a code-execution tool

Following feedback from the research community, the CritPt team and physics expert collaborators reviewed every challenge, corrected underspecified statements, errors, and typos, and improved the stability of the grader. We now run CritPt v2 agentically: models can write and run code in an offline sandbox with tools for computation.

We’ll be updating the Intelligence Index in late October, including changes to CritPt.

Key takeaways:

➤ Recent OpenAI releases top the leaderboard: GPT-6 Astra (max) and GPT-6.1 Sol (max) score highest at 83% and 82%, followed by Claude Opus 5.5 (max with fallback) at 78% and GPT-5.6 Sol (max) at 75%

➤ CritPt v2 shows the rapid progress at the frontier on research-level physics: GPT-5 (high), released in August 2025, scores 29% on CritPt v2, 54 points below GPT-6 Astra (max)

Thank you to @MiniHui_zhu, @MinyangTian1, @haopeng_uiuc and the broader CritPt team for their work on this revision and their continued collaboration

[Minhui Zhu @MiniHui_zhu]

@dlouapre Thank you for your contribution, David! We also adopted the expert-in-the-loop review framework: expert insights + high-performing model outputs yields much higher-quality content than fully automated review, while being faster than purely human review.

[Minhui Zhu @MiniHui_zhu]

Cross checking Astra’s response is only part of our review. We spent more time reviewing the full list that Anthropic sent us first, which likely involves Claude’s input. Our expert also independently conducted human review and cross checking with models of their choice before seeing Astra’s answers.

[Minhui Zhu @MiniHui_zhu]

@DrBeavisAI This work only took a subset of CritPt. These authors revised the problems in a different way and used very different evaluation setting, which effectively created a different benchmark based on CritPt.

[Minhui Zhu @MiniHui_zhu]

Our turbulence blob paper made the cover of PNAS! https://t.co/gGPCGf2Ra7
Photo by the amazing @takumi_matz

[PNASNews @PNASNews]

In this issue: A parasite drives movement in infected red blood cells, electric charges can accelerate without emitting radiation, and stratospheric methane loss. In PNAS: https://t.co/PeJOXYdai1

[steve hsu @hsu_steve]

In my experience GPT, Gemini, and DeepSeek v4 are strongest in theoretical physics. This benchmark suggests the same.

CritPt evaluates language models on solving unpublished, frontier-level physics problems that require genuine research-scale reasoning. The benchmark comprises 71 challenges (70 test challenges and one example), created by over 50 active physics researchers across 30 institutions and spanning 11 physics subfields

[Sam Altman @sama]

@hsu_steve mogging

[Minhui Zhu @MiniHui_zhu]

@Jess_Riedel @MinyangTian1 7/7 Final note: CritPt is a community effort from many physicists and we’re actively expanding it. If you (or any fellow physics friends here) would like to contribute problems to represent your research, we’d love to hear from you!

[Minhui Zhu @MiniHui_zhu]

@Jess_Riedel @MinyangTian1 6/7 Diversity: The example problem is a technical calculation in the computational flavor, but our full set includes a range of problems, including conceptually challenging ones. This diversity reflects the varied reasoning demands across research areas/groups.

[Minhui Zhu @MiniHui_zhu]

@abdurrahmanregi @MinyangTian1 Big question in the long term: if models eventually match this collective entry-level research capability (reliably solving all CritPt problems), can they make interdisciplinary discovery beyond human reach? Or at least help physicists across fields collaborate more efficiently?

[Minhui Zhu @MiniHui_zhu]

@abdurrahmanregi @MinyangTian1 LLM vs human (4/5):
Challenges in CritPt are at entry-level grad school research. therefore, each can be easily solved by at least one research group, so if you imagine a collective of junior PhD students covering all subfields of physics, the “human ceiling” is essentially 100%.