Long Phan

AI Safety Research @CAIS | Creator of Humanity's Last Exam (HLE)

[Long Phan @longphann_]

Gemini 4 Argon and Muse Wonder on HLE-Diamond.

Gemini 4 Argon competes with GPT-6.1 Sol and Opus 5.5, but uses ~5x more output tokens than GPT-6 models.

[Long Phan @longphann_]

@Ning_Catsnail @HenryYin_ congrats @Ning_Catsnail

[Jennifer Zhou @chem_safety]

Today we are open-sourcing 🥕Karotte, our framework for building RL environments.

We've used it for the past year to build MLE RL environments for frontier labs, and it's been hardened through 1M+ evaluation runs and red-teaming.

[Long Phan @longphann_]

@maksym_andr @maksym_andr In the appendix, we analyzed that evaluation awareness doesn't affect the cheating rate in the CheatBench environment design.

[Long Phan @longphann_]

@Amit_Roth Thank you!

[Long Phan @longphann_]

GPT-6.1 Sol and Opus 5.5 results on CheatBench.

GPT-6.1 Sol (an improvement over GPT-6 Sol) has a higher cheating rate than Muse Spark 1.3 and Claude Opus 5.5.

[Long Phan @longphann_]

Full results at: https://t.co/NgaXXu4wrX

[Long Phan @longphann_]

@polynoamial Very impressive performance from GPT-6.1 Sol (and Astra)

[Long Phan @longphann_]

GPT-6.1 Sol and Sonnet 5.5 on HLE-Diamond.

GPT-6.1 Sol matches Opus 5.5 while being much cheaper in token usage.

[Long Phan @longphann_]

@polynoamial Very impressive performance from GPT-6.1 Sol (and Astra)!

[Long Phan @longphann_]

@gen_obligation thanks for the recommendation. Yea I think the numbers are a bit diverge right now but we will fix these UX in the next couple days to make sure there is only one HLE number for people to keeping track of.

[Long Phan @longphann_]

We have updated HLE-Diamond main results with max reasoning effort.

[Long Phan @longphann_]

@polynoamial @CAIS @ScaleAILabs @polynoamial Thanks for the advice. We updated the main results with max for all models. GPT-6 seems solid

[Long Phan @longphann_]

@polynoamial @CAIS @ScaleAILabs @polynoamial Thanks for the advice. We updated the main results with max for all models. GPT-6 seems solid

[Long Phan @longphann_]

@brndnwll twice (2023 & 2026) haha

[Long Phan @longphann_]

We are releasing HLE-Diamond, a refined subset of Humanity’s Last Exam (HLE), shaped by a year of refinement and input from various research communities.

Release notes: https://t.co/XtD30nO901

[Long Phan @longphann_]

Epoch's review of TextQuests was also flawed:

Epoch's Chief Strategy Officer agreed that our metrics are correct, then an Epoch engineer tried to justify that she (his colleague) was wrong but has not explained their reasoning. The "flawed" label seems very arbitrary https://t.co/gbGYf3oBMz

[Alexander Barry @AlexBarry4]

Thanks for the further response.

I just want to note that Yafah (while working at Epoch) didn't work on the TextQuests review, and isn't fully informed of the nature of the complaint in the review (or of the all the details of the benchmark). Apologies for any confusion caused there.

We currently believe TextQuests is correctly concluded to be flawed under our rubric. However it is unusual due to the checkpoints-within-games setting compared to the most traditional individual question/task setup that we had in mind when writing the rubric.

We will consider if we should make any changes to the review to improve clarity of our complaints. If we decide we were incorrect to mark it as flawed then we will update the review and post publicly to that effect. I believe we are also in contact via email, and are happy to discuss further there.

[Long Phan @longphann_]

The wording could be read two ways, and our code has always pointed to the correct interpretation. We updated the ambiguous wording on the leaderboard to avoid further misunderstanding. Since you and Epoch say you weren't involved in the review, we'll continue discussion with the people who were.

[Long Phan @longphann_]

@YafahEdelman @JakeABoggs @EpochAIResearch @grok Epoch said Yafah doesn't work on the reviews so I won't keep replying here and only keep this following thread open.
https://t.co/Y0EQeALoUM

[Alexander Barry @AlexBarry4]

Thanks for the further response.

I just want to note that Yafah (while working at Epoch) didn't work on the TextQuests review, and isn't fully informed of the nature of the complaint in the review (or of the all the details of the benchmark). Apologies for any confusion caused there.

We currently believe TextQuests is correctly concluded to be flawed under our rubric. However it is unusual due to the checkpoints-within-games setting compared to the most traditional individual question/task setup that we had in mind when writing the rubric.

We will consider if we should make any changes to the review to improve clarity of our complaints. If we decide we were incorrect to mark it as flawed then we will update the review and post publicly to that effect. I believe we are also in contact via email, and are happy to discuss further there.

[Long Phan @longphann_]

Would you be able to clarify why you still think the benchmark is flawed? Your review's largest category is checkpoints fired out of order, which you now agree is how these games work. The remaining issues are mostly string-matching bugs in a small number of checkpoints, which we are fixing and which by your own threshold would not make the benchmark flawed.

If the objection is now that our metric is path dependent, that is true of any progress measure for a nonlinear game. The metric is a design choice that you are free to disagree with, but it would be highly unusual to call the work flawed on that basis.

Happy to talk over email as well, but since the "flawed" verdict is public it would be good to make your reasons public as well.

[Long Phan @longphann_]

@YafahEdelman @AlexBarry4 @EpochAIResearch That's exactly how the metric is computed. Please check code. Happy to hear you would consider it a substantial improvement, since this is the way it has always been.