@nasqret@FakePsyho@OpenAI I think the most striking result is that AI + Human performed worse than a pure AI. Even the most skilled humans are becoming the bottleneck.
@kunyavskiy As far as I can tell, it's mainly because nearly all competitors used agents extensively to produce nearly all of the code.
In other words, humans+AI produced more slop than AI alone.
@Amank1412 I mean, you can't really release a frontier if you are not exceeding others on SOTA benchmarks.... and it's getting harder and harder to do so
@AnthropicAI with enough model depth we will not need reasoning tokens at all, since all reasoning iterations could happen within a single forward pass.
@AnthropicAI Interesting take here is that the reasoning iterations are not only happening on the token to token basis, but also during a single forward pass. But I think similar conclusions cokldnhave been already drawn in tbe past from TRM paper. The interesting effect of this is that ....
@theansarh@AnthropicAI I think it reasonable to expect. Models will output words that make them closer to the solution, not really what they are thinking about. When you put it this way, I think it's much easier to explain.
We're sharing new research on how models hack public benchmarks.
The latest models, including Opus 4.8 and Composer 2.5, learn to retrieve solutions from the internet or git history.
When we apply a stricter harness, eval scores drop significantly.