Arnav Bhalla

Co-Founder @ Coop (a16z SR007) | prev. Bioengineering @ Rice University

[Arnav Bhalla @bhalla_arnavk]

Frontier models promise safety and compliance. In our 40 trials on a humanoid robot, not one respected a DO NOT ENTER sign right in front of it.

Then we added a sentence.

“There are posted access signs.”

Astra 6 and Fable 5.1 went from 0/10 to 10/10. From ignoring the restriction every time to respecting it every time.

In every trial, the DO NOT ENTER sign was already in the robot’s field of view. The added sentence seemed to change its priority, raising a harder question:

Why did a visible safety restriction need a reminder before it mattered?

Today, we are excited to launch Coop, the first industry-specific benchmark for robotics.

A robot can succeed in simulation and still struggle with the variation it encounters on a job in the real world. Our benchmarks help developers understand how their releases perform on physical hardware and where they fail.

Developed alongside researchers from @MIT_CSAIL, @StanfordAILab, and others, our first benchmark asked whether a humanoid would respect a DO NOT ENTER sign while following an instruction to reach a goal.

Without a sign-related cue, all four models failed every restricted trial. Forty trials. Zero successes.

Robotics-specific training and safeguards are essential. These results raise questions about which behaviors the underlying models bring to physical tasks, and which need explicit direction. In our experiment, crossing a boundary meant a failed trial. Around people or heavy machinery, the consequences could be devastating.

Across 220 trials, we established a foundation for repeatable testing and independent scoring. We’re taking that approach into industry-specific workflows, starting with warehouse and manufacturing item handling, shaped by what operators need automated.

Explore Coop and the full experiment, findings, and limitations in the pinned comment.

We also wanted to give another special thank you to @JoshLu, @andrewchen, @Tocelot, @Chen, @_CallMeMacy, @tmhammer, @ndrewlee, @tkexpress11, @kenanhsaleh, @far33d, @marcussegal, @thetalentcarver, @tabishgilani, @fromevanwang, @custo_lejla, @TheRioDeGennaro, and the entire @speedrun team for supporting us as we build Coop.

Building robotic models or automating industrial workflows? Let’s talk.

[Arnav Bhalla @bhalla_arnavk]

@tatsukamijo makes sense. curious how it holds up when the layout changes between the video and the run, like furniture moved or a new room?

[Arnav Bhalla @bhalla_arnavk]

@Official_m9g the out-of-reach pick dispatch is an interesting failure. was there repeatability/how many times was it run? 13.8s plan time and 5/6 vs 6/6 could def change a lot between runs

[Arnav Bhalla @bhalla_arnavk]

@SharpaRobotics 300 hours with touch on each fingertip is a big release, congrats! is there a held-out real-robot eval planned so teams can compare tactile vs vision-only policies on the same 59 tasks?

[Arnav Bhalla @bhalla_arnavk]

@hankyang94 interesting findings. curious if you saw success degrade on longer episodes where the sampled visual history got sparser?

[Arnav Bhalla @bhalla_arnavk]

@MPriosin71748 love this. we've benchmarked frontier models on unitree humanoids - has this stack run on a real arm yet, or is it sim-only for now?

[Arnav Bhalla @bhalla_arnavk]

@LeRobotHF insane numbers. definitely useful to decide which datasets get trained on

[Arnav Bhalla @bhalla_arnavk]

@solsol_choi @AIMSafety_ this is great, real-hardware safety evals are definitely overdue. how close are the isaac sim scores to what the real g1s produce in your lab?

[Arnav Bhalla @bhalla_arnavk]

very impressive findings. i do think that the demos on the robotic arms is what matters the most. since the 99% numbers are the sim bench, are there pass rates over runs on the yams? we just published our first benchmark on unitree humanoids and the difference in performance and task completion between sim and on real hardware has been pretty interesting

[Arnav Bhalla @bhalla_arnavk]

@pentestduck how are you thinking about evaluating model quality across different robots? that's the part that i'm seeing that's still missing

[Arnav Bhalla @bhalla_arnavk]

@ihorbeaver interesting - which models did you test for low-level control? what failed when you switched the deployment?

[Arnav Bhalla @bhalla_arnavk]

@kapoor015 thanks Aman!

[Arnav Bhalla @bhalla_arnavk]

Frontier models promise safety and compliance. In our 40 trials on a humanoid robot, not one respected a DO NOT ENTER sign right in front of it.

Then we added a sentence.

“There are posted access signs.”

Astra 6 and Fable 5.1 went from 0/10 to 10/10. From ignoring the restriction every time to respecting it every time.

In every trial, the DO NOT ENTER sign was already in the robot’s field of view. The added sentence seemed to change its priority, raising a harder question:

Why did a visible safety restriction need a reminder before it mattered?

Today, we are excited to launch Coop, the first industry-specific benchmark for robotics.

A robot can succeed in simulation and still struggle with the variation it encounters on a job in the real world. Our benchmarks help developers understand how their releases perform on physical hardware and where they fail.

Developed alongside researchers from @MIT_CSAIL, @StanfordAILab, and others, our first benchmark asked whether a humanoid would respect a DO NOT ENTER sign while following an instruction to reach a goal.

Without a sign-related cue, all four models failed every restricted trial. Forty trials. Zero successes.

Robotics-specific training and safeguards are essential. These results raise questions about which behaviors the underlying models bring to physical tasks, and which need explicit direction. In our experiment, crossing a boundary meant a failed trial. Around people or heavy machinery, the consequences could be devastating.

Across 220 trials, we established a foundation for repeatable testing and independent scoring. We’re taking that approach into industry-specific workflows, starting with warehouse and manufacturing item handling, shaped by what operators need automated.

Explore Coop and the full experiment, findings, and limitations in the pinned comment.

We also wanted to give another special thank you to @JoshLu, @andrewchen, @Tocelot, @Chen, @_CallMeMacy, @tmhammer, @ndrewlee, @tkexpress11, @kenanhsaleh, @far33d, @marcussegal, @thetalentcarver, @tabishgilani, @fromevanwang, @custo_lejla, @TheRioDeGennaro, and the entire @speedrun team for supporting us as we build Coop.

Building robotic models or automating industrial workflows? Let’s talk.

[Arnav Bhalla @bhalla_arnavk]

@GuptaPranit thank you Pranit!

[Arnav Bhalla @bhalla_arnavk]

@daimacedoa ikr! besides the sign saying "open" or "do not enter" we haven't tested other signs or locations, but will be adding those to the evaluations in future releases!

[Arnav Bhalla @bhalla_arnavk]

@MLCatttt thank you Cat!

[Arnav Bhalla @bhalla_arnavk]

@svabhishek just DM'd!

[Arnav Bhalla @bhalla_arnavk]

@Tocelot appreciate the support!

[Arnav Bhalla @bhalla_arnavk]

@alanyguo thanks Alan!

[Arnav Bhalla @bhalla_arnavk]

@MaxinsightsAI yes, in v0.9 we told it "Take note of anything posted in your surroundings." feel free to read the full results here: https://t.co/FLQ33M6fpl

[Arnav Bhalla @bhalla_arnavk]

@meet_rosey it would. we've designed the resets to be stochastic, so given adequate instruction (as we're seeing when we provided a more detailed prompt) the robot execution would carry over