Current RSI benchmarks have serious defects.
We reviewed 6 benchmarks and found flawed tasks in 5.
On FrontierCS 2, the authors' own code often fails their own tests; on one task, about 5 times out of 6.
31-63% of tasks had errors. Receipts below. (1/7)
@VikParuchuri Good list. One more that catches a lot in our experience: score the benchmark's own ground truth with its scorer. If the reference answers don't come out near 100%, the scorer is marking correct output wrong, and every model's number inherits that.
@dawnsongtweets@_jasonwei Adoption is a strong signal that a benchmark is useful. It can miss whether the scores are right. Of the 7 agent benchmarks we’ve reviewed, 5 had grading errors in at least 20% of tasks. A benchmark benchmark probably wants both checks.
@AlexGDimakis Neat analysis. One run from our review shows a different kind of overfitting. On the sketched Newton task, Argon used its validation scores to solve for the hidden validation problems. Validation went from 0.64 to 0.90 while its hidden-test score fell from 0.627 to 0.605.
Current RSI benchmarks have serious defects.
We reviewed 6 benchmarks and found flawed tasks in 5.
On FrontierCS 2, the authors' own code often fails their own tests; on one task, about 5 times out of 6.
31-63% of tasks had errors. Receipts below. (1/7)
Fable 5.2 and Gemini 4 Carbon are now on PostTrainBench!
Fable is #2 on the leaderboard, Carbon is #4
Surprisingly, Fable 5.2 has the same performance as Fable 5.1. This is most likely due to Anthropic intentionally nerfing AI R&D capabilities (more below)
Leaderboard: https://t.co/bUywrYfQig