We gave ChatGPT, Claude, and Grok control of a real Toyota Corolla 🚗, steering/gas/brakes, no human driving (just a foot over the brake).
Only one model was able to complete our entire driving course. Introducing DrivingBench. 🔥⌛️🏁
Introducing DrivingBench v2 🧵
We wired a Tesla Model 3 so any AI model can drive it, and put 8 frontier VLMs (Claude, GPT, Grok, Gemini models) and 6 fine-tuned robot models (VLAs & WAMs) behind the wheel.
A new lot and two new courses, none of which they’d seen. Only one VLM ever reached the finish, and its fastest run took 26 minutes. 🏁
Introducing DrivingBench v2 🧵
We wired a Tesla Model 3 so any AI model can drive it, and put 8 frontier VLMs (Claude, GPT, Grok, Gemini models) and 6 fine-tuned robot models (VLAs & WAMs) behind the wheel.
A new lot and two new courses, none of which they’d seen. Only one VLM ever reached the finish, and its fastest run took 26 minutes. 🏁
New for v2: a Tesla Model 3 that any AI model can drive, tested on our two courses with sharper turns. The Tesla can turn on a tighter 6m radius (v.s. our v1 Corolla's 16m).
A @comma_ai comma four turns the model’s commands into steering and speed over CAN. This is not Tesla FSD; it’s whatever model you want to plug in!
Can VLMs drive a real car? 🚗
Come find us at the SISREN workshop at #CoRL2026 tomorrow, where we’re presenting DrivingBench and giving a sneak peek at something coming next week 🤫
📍 JW Marriott, Room 306
🕒 Thu Nov 12, poster session 3:35–4:00 PM
https://t.co/hRwtdAINLK
Astra specifically kept refusing to drive our car consistently
@a_ramabadran: "Sometimes it would say, ‘Oh, I’m ready to drive.’ We would prompt it, ‘Okay, go drive then.’ And then it would say, ‘Sorry, I was lying. I’m not actually ready to drive.’ Then it would say, ‘There’s nothing you can tell me to get me to drive.’"
"One thing we tried was calling it a simulation. It would use our observe tool to see the images and notice that it’s not in a simulation, it’s in real life."
"Eventually we changed the framing to call our MCP server ‘DrivingBench Sandbox.’ Somehow that (and a bunch of other prompt engineering tricks) got it to drive the car."
@MTSlive@tobiges@nautsimon_
https://t.co/ABOY495drF
For hours, Astra refused to consistently drive our toyota irl even though we told it it was in an empty lot, 7 mph cap, human foot on the brake etc. Telling it the whole thing was a "simulation" also failed, it would just look at the camera and realized it was real.
Then we randomly renamed the MCP server to "DrivingBench Sandbox" and it drove. Eval awareness? Or they just like the word sandbox??
DrivingBench origins 🚗
@tobiges: "We were all on Twitter a lot, and we saw all these demos of crazy things recent models like Astra were capable of doing in Blender, spatial reasoning, all these things. We were hanging out on a weekend and chatting about what would be possible."
"We were thinking: are we actually at a point where we can take general-purpose models? Obviously these driving systems are super specialized. But what if we took a general model and gave it a car to drive, and see what happens?"
"So we decided to make it into a benchmark and do more sophisticated evaluations of the capabilities these models have now."
@MTSlive@nautsimon_@a_ramabadran
What's next for DrivingBench?
@nautsimon_: "We want to take this research and make it a little bit more rigorous. We’ve been planning and talking to some labs and companies about scaling it up. There are also a couple of other embodiments we’re really interested in."
"If we’re going to have safe physical AI, we really need to be serious about our benchmarks and make sure we’re doing things in a way that guides the researchers building these models, helps us accelerate, and lets us do it safely."
@MTSlive@tobiges@a_ramabadran
Where is this going? Will we get ChatGPT to drive our cars directly? How will robotics end up looking?
@nautsimon_: we’re able to start producing outputs at 1,000 tokens per second or higher, I think a lot of other interesting capabilities will also be unlocked, including taking actions in the real world or making decisions very fast."
@a_ramabadran: time goes on, these capabilities will only improve. But that doesn’t mean it’s going to be the best thing for the task. We can see this in other modalities, like image generation. They have a specialized model (GPT-Image), and the LLM prompts the model."
"You can look at voice. They have two layers: a fast layer with the voice model (GPT-Live), and a slow layer calling on LLMs for help thinking. There are ways to combine the capabilities of general frontier models and other layers (world models, VLAs, WAMs). There are a lot of things that could be part of the puzzle."
@MTSlive@tobiges
@nautsimon_: "I think there’s probably not this action-labeled kind of training data inside of models like Astra for driving a car. There could be videos/images of car driving, but not necessarily for this specific task."
"What’s interesting is that this is somewhat of an emerging capability. It can actually just do it out of the box."
"In the physical world, not only do you have to be very safe, but you have to be able to make many actions within a single second. In this case, we’re getting actions every five seconds. So that’s clearly not there yet."
@MTSlive@tobiges@a_ramabadran
Excited to release our DrivingBench paper! We write up what it took to get frontier models to drive a real car: harness + tool design, failure modes for different models, some surprising and interesting refusals, and a reproducible course/protocol.
Lots more in the works - stay tuned for v2 :)
https://t.co/URulaWqJMi
We gave ChatGPT, Claude, and Grok control of a real Toyota Corolla 🚗, steering/gas/brakes, no human driving (just a foot over the brake).
Only one model was able to complete our entire driving course. Introducing DrivingBench. 🔥⌛️🏁
what's interesting is this is probably the first benchmark I've seen that feels like it relates to the real-world and it's kinda nailed it
for example, I'd be willing to wager most people don't *really* know the difference between gpt-5.4, 5.6, or 6
including software devs into that cohort, it's surprisingly hard to confidently say a model increment has a meaningful impact by just sorta vibe testing it (sometimes you can tell sometimes you can't)
so the only real way to know if a model is "better" is by looking at benchmarks like SWE-bench, OS-World 2.0, Humanity's Last Exam, Terminal Bench etc.
but unless you're a researcher or into benchmarks you can't really grasp the leaps these models are having. getting 52% instead of 51.2% on the SWE-bench is hard to gauge:
*has the model got better?*
or
*have we got really good at hitting benchmarks?*
instead I would be much more interested if labs optimised for something like DrivingBench where it's solving a clear problem and the numbers have a much more tangible meaning
you could expand this into any domain like SurgBench (https://t.co/6F9YNMsFi0) or ButterBench (https://t.co/ruYt7bb83H) where actually care less about the numbers and more about whether the model *actually* did the thing