@madiator matches what we've seen using opus 5.5 agent teams in claude code - honestly mindblowing when collaborating on research tasks and i think its because they proactively message each other and read them. curious if duplicated work drops if you force a read before every claim
@AlexGDimakis the sol result is wild. how can it send 3.4x more messages and do worse than a team that doesn’t send messages at all haha. it read less so maybe communication != coordination, listening seems to be the actual skill
some support for this in @bespokelabsai idea basin data: gemini models were the least likely to get stuck on their first idea (31–34% of runs vs ~60% for some others, though intervals are wide). doesn’t tell us if the ideas are different from claude/gpt’s though, would love to see cross-model basin overlap
@kellyhongsn the gemini one is wild. it ran the exact experiment showing the variants weren’t OOD and then just ignored the result. feels like the bottleneck isn’t gathering the information, it’s actually updating on it
yeah exactly! and in our runs they seem to misjudge it in a consistent direction by overweighting how hard the idea is to build vs how much they’d learn from trying it. would be cool to add costs to re-search, like some questions being more expensive, and see if models still go for the high-info ones. happy to share some traces if useful!
super cool setup! the generation-not-judgment result is interesting bc in our agentic runs we see the reverse a lot: the model writes the better idea in its reasoning, decides it's too complicated, and never tries it. maybe once execution costs enter the picture, judgment becomes the bottleneck again?
love this!! my guess on why new ideas pay off so much more than refining: a new idea usually means the agent actually understood something new about the problem, and refining mostly doesn’t. we’ve seen runs where the score keeps climbing for hours while what the agent understands stays totally flat
the evidence notes + working state split is nice! we’ve seen agents’ understanding drop right after compaction, and recall questions mostly test whether facts survived rather than if the agent still knows why things happen. curious if you tried keeping a short ‘current hypotheses’ section, or testing whether it can still predict outcomes after compacting
@ai2face held-out tests help, but the stronger signal is rewarding what the agent can predict. if it actually understood why something worked, it should be able to call the outcome of experiments it hasn’t run yet. noise can’t fake that
we’re about to repeat this with AI scientists. agents get rewarded for pushing one number up, and an agent that hill-climbs a benchmark looks rigorous the same way p<0.05 does
A full generation of young scientists have grown up thinking the p-value is the way to measure scientific rigor. That is a collective failure of our scientific training mission. It's much easier to teach, but it has had a very negative effect on how young scientists think.
@maksym_andr worth noting METR’s own read here: they flag them as possibly broken tasks or fat-tailed difficulty, not a missing capability. and model 4 is above 50% on every benchmark, so a flat line looks more like a ceiling than taste. would be a great taste benchmark if it’s real though
@Ronalfa similar take: most ML research evals aren’t research evals, that’s just coding and hill-climbing a metric. we do need models to do research though, and that means picking the question and figuring out why something worked
@DimitrisPapail sol’s switches do mostly go to new ideas, which is cool. but it also submits ~3x as often as opus and overfits way more, astra beats it on test (0.60 vs 0.51) with a quarter of the overfitting gap. creative explorer, sloppy refiner maybe