shaped

post training data for rl @

[shaped @shaped]

AutomationBench Verified is out.

We went through all 600 public tasks in @Zapier's AutomationBench and checked every verifier.

• Agents flagged 323 verifiers as suspicious, human review confirmed 206 real bugs, all 206 are fixed.
• We replayed 1,235 Kimi K3 runs on the old and fixed verifiers and 344 of them (27.9%) got a different grade
• Where verifiers were too strict, pass rate went from 18.8% to 43.8%
• Where they were too lenient, it dropped from 60.2% to 49.7%

Audit and dataset links below.

[shaped @shaped]

@unusual_whales jensen recruiting harder than every lab’s poaching team combined💀

[shaped @shaped]

@ewveggies 😭😭😭

[shaped @shaped]

@ssh_vom godspeed

[shaped @shaped]

@witcheer a 27b on a single 5090 doing this lowkey kills the “you need a cluster” argument lol

[shaped @shaped]

@dejavucoder crazy how benchmark auditing has become a regular thing now lol. its nice to see people keep up the standards. super cool work!

[shaped @shaped]

@yzeng58 lowkey just wanna see what the LM thinks the whales are saying lol

[shaped @shaped]

@matthewnoto73 real

[shaped @shaped]

@SeanLiu0x60 @dzy97 pure pass/fail feels so wasteful once you’ve seen what a judge can tell you about why it failed lol

[shaped @shaped]

@XiaotianHan1 congrats on the release! which stage was the most painful to get stable, curious if it was RL or the data side

[shaped @shaped]

@supremeZhuang congrats! curious what the biggest tell was this cycle when a review was mostly AI written

[shaped @shaped]

@an_vo12 would reviewers actually show up though? feels like the ones who didn’t read the paper are the same ones who’d ghost the call lol

[shaped @shaped]

@sk_2496 biggest gap i keep hitting is partial-credit rubrics that let agents ship shortcut solutions that still score

[shaped @shaped]

@sulaiman_svesal self-rewriting harness plus rl on leftover hard tasks is interesting. have you seen the new harness introduce failure modes that the original verifier then misses?

[shaped @shaped]

@rohanpaul_ai makes me wonder if agents need regression tests for their own memories now lol

[shaped @shaped]

@doesdatmaksense pretty useful research, especially since cheap evals are only useful if you can trust the scores

[shaped @shaped]

@MTSlive how'd they validate the phages before using them clinically

[shaped @shaped]

@ryan_punamiya really nice work, getting annotation costs down this much could open up a lot more RL experiments

[shaped @shaped]

@stevehou kinda wild that generating candidates is getting easier while testing them still takes forever

[shaped @shaped]

@AtaeiMe their marketing team is the github repo

[shaped @shaped]

@kaleybrauer lowkey sounds like a way to make a model forget stuff without making it dumber