Vibrant Labs

Autoscaling RL environments and benchmarks

[ikka @Shahules786]

1/n

In last week’s Paper Club, we focused on three papers that give insights on how to generate post-training data that meaningfully improves agent capabilities. This is the central question that @ VibrantLabs exists to solve.

- PlanBench-XL (from @UofIllinois), which focuses on realistic tool envs
- TMax (from @allen_ai, and @uwcse), which focuses on synthesizing and training against harder tasks, and
- Autodata (from @AIatMeta), an agent data scientist loop that synthesizes data and can be meta-optimized.

[ikka @Shahules786]

We had a blast last week when we hosted the first Hot Takes game night (+ dinner) focused on autoscaling RL envs. Carefully chosen guests were instructed to bring the most controversial opinions they could to a discussion on post-training.

As per usual, only technical practitioners, no VCs.

We had conversations on how to improve diversity of synthetic envs, distribution collapse, whether it’s even possible to do entirely autoscaling and RSI.

We host these specific research-focused events regularly, with different focuses and activities. If you’re working on any of the following problems, we’d love to include you in the next one:

- Unsupervised environment design
- Efficient RL training for multi-turn tool use
- Self-evolving benchmarks
- Autonomous AI research
- Open-Endedness

[ikka @Shahules786]

At @VibrantLabsAI, we’ve always been a research-minded team internally, so it felt completely natural when we started doing a regular, organized Paper Club as a team.

What we didn’t expect was how much interest we’d get from that over the past few months from folks outside the team.

Every Paper Club, we post our notes from the discussion on Twitter/LinkedIn and mention the authors and the orgs involved with relevant papers.

We often end up speaking with those authors before and after our discussion, and nowadays, we even work with some of them on a regular basis to help us autoscale RL envs.

This week, we decided to formalize our Paper Club a little further by adding a dedicated section to our site where you can see all of the notes and papers discussed: https://t.co/xqXvYw4LO7

Hope anyone who is following along enjoys it.

[Kyle Jeong @kylejeong]

@VibrantLabsAI evals at scale with @browserbase sounds good to me

[Jithin James @real_jjmachan]

we are releasing ecom-bench which tracks how agents perform in basic e-commerce tasks with DOM vs CUA modalities with the stagehand harness from @browserbase .

there were a couple of interesting takeaways

[ikka @Shahules786]

Everyone building browser agents eventually comes to the same divergence: should the agent read the DOM or look at the page visually like a human?

At @VibrantLabsAI, we ran this experiment on Ecom Bench (with 40 verified shopping tasks on live storefronts) and the answer was somewhat complicated.

Labs like @yutori_ai and @AnthropicAI are thinking deeply about the trade-offs between DOM and CUA (see their "bitter lesson for web agents" and “demystifying evals” posts, respectively).

Full results, cost and latency breakdowns, and the failure-mode analysis are in the thread below and the post.

Blog: https://t.co/2w22E17oW1

Env on @PrimeIntellect (built w @browserbase): https://t.co/6B89dW7BbC

[justine @justinemach_]

static benchmarks can't tell you if your agent can actually do human tasks online. @VibrantLabsAI built one that can: their web agent eval runs on live shopify stores + deterministic verifiers via @browserbase. run it yourself↓

[Dheeraj Mohan @McDhee98]

Evaluating web agents on the actual web is hard. @VibrantLabsAI did it right: live Shopify stores, deterministic verifiers, all running on @browserbase. Open for everyone to run now ↓

[Miguel @miguel_gonzf]

@VibrantLabsAI Less saturated benchmarks that showcase the gap in current model capabilities are necessary to continue pushing the frontier

Love this from @VibrantLabsAI with browserenv, excited to see more benchmarks in different verticals!

[Vibrant Labs @VibrantLabsAI]

14/

We build these environments so that evaluating and training web agents on real, verifiable tasks stays open to any lab, regardless of who owns the harness.

If your team is interested in the post-training data pipelines we work on at @VibrantLabsAI, you can reach us at [email protected]!

[Vibrant Labs @VibrantLabsAI]

1/n

For browser agents, a major bottleneck in evaluation is truthful scoring on the live web. A task is only as good as your ability to confirm the agent actually did it, on a real site whose state keeps moving and that the agent can potentially misreport.

So we took matters into our own hands.

Today, we're releasing Ecom Bench on @PrimeIntellect: 40 shopping tasks on real Shopify storefronts, each run in a live @browserbase browser and graded by a deterministic verifier.

https://t.co/NJ43WnTk4O

[Vibrant Labs @VibrantLabsAI]

13/

Ecom Bench is available now for training and evals:

prime env install ecom-bench

DOM mode runs the full 40-task set, and grounding="cua" flips the same tasks to pixel grounding for the comparison above.

@PrimeIntellect (built on @browserbase): https://t.co/hyvoa5oW9B

Full writeup: https://t.co/8KocqyIJ7y