Seth Schilbe

Building RL environments for long-horizon credit assignment

CTO @oroagents (YC F26), prev senior @awscloud @amazon

[Seth Schilbe @SethSchilbe]

Autonomous businesses operating in the real world - coming soon

[ORO @oroagents]

The economic value of agents will soon come from running whole autonomous businesses, not just from finishing one-off tasks. When you run a business, the result of your own decisions arrive weeks later. You have to notice them, trace them to something you did, and act before they compound.

Today, we're excited to announce SMB Bench, where we measure all the frontier models' ability to observe changes in the environment and adapt accordingly.

[Seth Schilbe @SethSchilbe]

@matiasdrejer @agenticdataco Who's doing the voiceover? 👀

[Seth Schilbe @SethSchilbe]

@mcgeedan42 This is some Sci-Fi level CI magic! What do you mean you guys can just continually fix our tests to make it faster and faster?!?

[Seth Schilbe @SethSchilbe]

We have tried a few ablations of the configuration. We have tested several models with larger notes but have seen no or very limited performance improvement.

What is interesting is that Gemini Argon is the only model that consistently gets the note saving wrong, and frequently doesn't save notes on errors, even when it has a lot of tool calls left.

[Seth Schilbe @SethSchilbe]

Going so deep into model behaviour has given us a really unique perspective on the exact capabilities that the current models lack. Lots of tokens and late nights later, we see these things even with the latest models:

1. The notice-act gap - Similar to the findings in https://t.co/nyLDM5apLz we see that the models are able to notice information and even predict a strategy that would increase performance, but fail to take the actions necessary to actually implement that strategy. We even see "experimentation" in notes/reasoning, but that doesn't work well without actually implementing the experiments. 3 out of 26 times that Astra's notes mention buying more to close the gap, it buys the same or LESS for that day.

2. Impact on the environment - The frontier models consistently underestimate the impact of their actions on the environment and don't recognise the second-order effects. In SMB Bench we see that models consistently treat a short delivery to a customer as a one-day cost. A customer who gets shorted quietly orders less from then on. Across the runs where this happens, a model connected its customers' falling orders to its own short deliveries in only ~21%, and in just ~17% of those did it then protect every account within three days.

Noticing a problem, changing behaviour and actually improving the outcome turn out to be three separate skills that no frontier model has gotten correct in combination in these long horizon tasks.

[ORO @oroagents]

On SMB Bench, Claude Fable 5.2 at high reasoning effort is the second model to beat our naive buying rule.

It still shows the gap we see in other models, to a lesser extent. It identifies that shorting a customer costs future orders, and it acts on that, but only partially: it protects about half of the shortfalls it could, and its customers still end the quarter ordering far less.

It also underweights second-order effects. It recognises that its own decisions change how customers behave, but in several runs it only acts once its own records have confirmed the damage, weeks after it began and weeks after the necessary information is available.

[Seth Schilbe @SethSchilbe]

@uxduck @UXDuckHQ @ycombinator Congrats Ben!!

[Seth Schilbe @SethSchilbe]

@kaiqu_liang @AndrewLampinen @cocosci_lab @jaime_fisac Curious about Opus-4.8 masking dropping from 0.41 at k=10 to 0.18 at k=15. Did the longer notes start scoping the rule back down, or is that noise across six sequences?

[Seth Schilbe @SethSchilbe]

We are updating the full report as more models are released:
https://t.co/vih2mlnm89

[Seth Schilbe @SethSchilbe]

@KlausCodes Is this how you are announcing that you guys are also now a sandbox company?

[Seth Schilbe @SethSchilbe]

@alexandr_wang Curious to see if a mixture of models would bridge some of the gaps we are seeing with the family of Muse models on our benchmark

If Wonder does the planning and Ember does the work

https://t.co/vih2mlnm89

[Seth Schilbe @SethSchilbe]

@alexandr_wang https://t.co/vih2mlnm89

[Seth Schilbe @SethSchilbe]

On our recently released benchmark that tests models running a wholesaling business, the performance is in line with GLM 5.3, Grok 4.7, and Muse Spark 1.3.

Interestingly, Mused Wonder 1.0 is still the least performant.

Working on Luna & Haiku evaluations for true comparison.

[Seth Schilbe @SethSchilbe]

@AIatMeta Full report and breakdown -
https://t.co/vih2mlnm89

[Seth Schilbe @SethSchilbe]

@AIatMeta On our recently released benchmark that tests models running a wholesaling business, the performance is in line with GLM 5.3, Grok 4.7, and Muse Spark 1.3.

Interestingly, Mused Wonder 1.0 is still the least performant.

Working on Luna & Haiku evaluations for true comparison.

[Seth Schilbe @SethSchilbe]

@finkd Full report and breakdown - https://t.co/vih2mlnm89

[Seth Schilbe @SethSchilbe]

@finkd On our recently released benchmark that tests models running a wholesaling business, the performance is in line with GLM 5.3, Grok 4.7, and Muse Spark 1.3.

Interestingly, Mused Wonder 1.0 is still the least performant.

Working on Luna & Haiku evaluations for true comparison.

[Seth Schilbe @SethSchilbe]

@viswa_kotra Lets go!!

[Seth Schilbe @SethSchilbe]

@nbushak @boxesdotdev Congrats, guys! Love the product

[Seth Schilbe @SethSchilbe]

@Peak__Twilight @0securitylabs @ycombinator Lets go!

[Seth Schilbe @SethSchilbe]

Do you guys need a benchmark for this? I know a guy
@jonathangrahl @NirvikBaruah @AniC_dev

[Seth Schilbe @SethSchilbe]

My entire feed right now

[Anton Osika @antonosika]

Now I wanna see the showdown CC @akshat_b @mlejva @rauchg