idler

Hill-climb any benchmark, for any model, in any domain.

[idler @idler_ai]

ShelfLife E-Sim is our second public collection, built on the same data foundation as ShelfLife.

Full results and the complete set of environments are available now: https://t.co/CuEpZLRAVd

[idler @idler_ai]

We ran all rollouts on each model’s preferred harness (Grok Build for Grok, Claude Code for Opus and Fable, etc). Then, we reconstructed over 500 rollouts, event by event.

Grok 4.6 performed the best with a score of 0.66, but only realized 30% of all the profit potential, where we define 100% to be if it knew the future customer demand. Four observable behaviors contribute to its superior performance: moving/purchasing stock optimally, recovering from errors in tool calls, and using turns efficiently. We conclude this by comparing Grok’s estimated average-reward advantage with and without accounting for these four behaviors: the advantage falls from +0.107 to +0.022, and the adjusted estimate is statistically indistinguishable from zero (95% CI: −0.036 to +0.088).

We also observed behavioral differences between runs. For a task in the collection, Grok chose to operate at a much higher cadence, transferring inventory on 89/120 days with only 1.7% of tool calls resulting in errors. On the same task, Fable transferred inventory on 20/120 days and hit tool call errors at a much higher rate, 15.1%.

A simple greedy heuristic that uses a 90-day forecast beats all 6 frontier models tested in one out of four runs. 12% of the time, a SOTA model earned less money than a manager that does nothing.

In 38% of failed runs, the model lied about being successful. In these cases, the agent lost money for 4 months and then reported that things went well.

A human manager who did that would be fired.

[idler @idler_ai]

Most business simulations generate demand from a statistical model. Once an agent can reverse-engineer the model, it can plan against it, and its score doesn’t reflect how it would do in reality.

ShelfLife E-Sim replays real customer orders instead. The demand is driven by years of customer orders, from an actual retail chain, with real-world irregularity. Similar to real life, ShelfLife E-Sim features random events, delayed effects, and irreversible actions, under a fixed turn budget. Suppliers can arrive late, sold units can be returned, and a purchase order can be postponed.

Each simulated day, the agent must react to customer orders and decide what to restock, when to order it, and which warehouse to store it.

ShelfLife E-Sim is a follow-up to our ShelfLife-200 collection, which was built to answer the question: can you deploy an agent in a leadership role at a company and trust it to act autonomously?

If an agent makes a mistake, it must deal with the consequences weeks later, the same way a real store manager would.

[idler @idler_ai]

We cloned 10 years of sales, inventory, and supplier data from a retail chain into a business simulation. Then, we put frontier AI models in charge for 120 days.

Most made less money than a simple greedy algorithm

Introducing ShelfLife E-Sim: https://t.co/CuEpZLRAVd

[jj @jjzhao__]

determining the construct validity and base quality of RL envs is pretty trivial. the sheer quantity of slop and unverifiable synthetic content circulating in the market is ridiculous. there is a gap between the rl data being sold vs the truly correct/efficient/scaleable design. a byproduct of this is that we are far from being able to deploy agents in higher-level management layers autonomously.

if you believe in AGI/RSI, turning humanity’s raw data into learnable, high-signal rewards is clearly one of the most important problems in the data space. it is also extremely difficult. new RL companies appear every day because the problem remains unsolved/shit tons of cash will fly around until it is. as an RL company, if you aren’t careful, you’ll be too busy leaping for bills to realize you never understood the problem. there’s a ton of noise/talk about the space rn, and tbh its a result of no party fully knowing what is going on (it's kind of impossible). i personally see it as a fascinating little prediction game.

im bullish on the space, not because of the market cap, but because this is an extremely difficult and often frustrating engineering/anthropological project. its never the best time for anything, but working with the @idler_ai team on rl reminds me how exhilarating it is to do hard things. kind of like hand drawing tessellations well...

i remember the philosophy behind creating idler’s very first env - provable correctness, adjudicated fairly, and truly difficult for the frontier coding model at the time (churned out by a bunch of guys in an apartment who learned how to code before AI existed). we used real data then, and we still use real data now. time has passed, but we’ve kept our principles in developing new research directions and have successfully scaled. while the battle to keep up with the frontier gets harder every week, we’ve managed to continuously exceed sota model capabilities with innovating truly difficult envs, not arbitrary reasons to fail models.

be that as it may, seems like we are no longer underground, so happy to discuss our philosophy/work (im in the bay/sf!). leading the eng development of the ShelfLife collection in particular has left me deeply distraught at times, but it has also given me a sliver of understanding into what AI can become for human development. extremely grateful to run this race w @chubivan, @naluconcepcion, @cha0sg0d_, @notdavidhuang, @matthewhansen03, and the rest of team. seems like we are going to make a trillion good envs.

ps our simulation and law datasets are quite sick too, access upon request :)
https://t.co/Hyc1j2wweB

[idler @idler_ai]

We're just getting started, and we're hiring. Join us – [email protected]

[idler @idler_ai]

Today, we're introducing idler – a frontier data research lab.

We've raised a $9M seed round, led by @0xalpo and @FrankieIsLost from @paradigm, with participation from @ycombinator, @LongJourneyVC, and angels including @marrc, @catheryn_li, @dcposch, @JanineLeger, @LynettCapital, @feross, and @nur_daulet_.

This funding helps us build the evals, benchmarks, and reinforcement learning environments that the world's leading frontier labs use to train and measure their models.

[idler @idler_ai]

ShelfLife, our first public benchmark, is out today — a digital twin seeded with the complete operational data of a live, profitable, growing multi-brand retailer running 9 stores, 3 commerce platforms, and a multi-supplier sourcing operation. Post-training Nemotron 3 Nano on 41 ShelfLife tasks raised its Finance Agent Benchmark score from 30.0% to 41.8% (p ≤ 0.001); full results and sample tasks are available upon request. https://t.co/ZI6WQCgp3s