Sarah Chitty

Software Engineer at ScaleAI

[Cua @trycua]

1/ Today we're launching Cua Driver Bench, together with Scale AI's AgentEnv.

Can an agent do a desktop task in the background while you keep working undisturbed? On macOS, Cua Driver fully passed 58%, Codex computer use 33%, Vercel's agent-desktop 20%, browser-use 3%.

[Revan @RevanAleti]

How far can a better harness take a lightweight model?

Using Agent Env Framework from @ScaleAILabs, we let Haiku's harness recursively improve across 3 games and put it head to head against Opus.

See more at: https://t.co/LZkKRqLYaS

[Sarah Chitty @sarahchitty22]

https://t.co/WDuJntVJay friday afternoon viewing party

[Guts @agutmanstein]

With @huggingface, we made AirOpsEnv: real storm days across JetBlue's network, where an agent decides which flights to cancel or hold while FAA ground stops and lightning ramp closures hit JFK and Boston.

It's all open source and built from public FAA, BTS and NWS data: 94 storm days, each graded against the best plan possible in hindsight. On those days JetBlue cancelled 5.6% of its flights, and the hindsight plan cancels 2.2%.

AgentEnv Framework makes that easy: a whole day, with news arriving when it was actually issued and a virtual clock, is 7 steps in one task.json.

pip install agentenv-framework

[Sarah Chitty @sarahchitty22]

@ScaleAILabs You can run phone use agents on real devices yourself through our open-source platform AgentEnv using the @Mobilerun plugin!

Check it out here
https://t.co/EFBPcxTHfG

[Sarah Chitty @sarahchitty22]

Most phone agents are tested in simulated environments, but the real world is messier: carts persist, listings change hourly, and navigation varies.

At @ScaleAILabs, we ran agents on real phones with real apps on AgentEnv Framework.

See more at https://t.co/os4LJLyTPG

[Sarah Chitty @sarahchitty22]

@ScaleAILabs Agents struggle on capabilities humans find intuitive. Whether its navigating to a price filter, manipulating a calendar widget, or keeping track of requirements, even frontier models get tripped up.

[Sarah Chitty @sarahchitty22]

@ScaleAILabs A live app has no reset button: whatever changes an agent makes -- a full cart, a saved search, a wishlist item -- is waiting for the next run. So after each task, a janitor agent cleans up the agent's actions before the next run.

[Edgar Arakelyan @earakely]

https://t.co/6cEndYX6xj