DrivingBench

frontier LLMs drive a toyota corolla -

[DrivingBench @DrivingBench]

We gave ChatGPT, Claude, and Grok control of a real Toyota Corolla 🚗, steering/gas/brakes, no human driving (just a foot over the brake).

Only one model was able to complete our entire driving course. Introducing DrivingBench. 🔥⌛️🏁

[Charlie @charlie_m_day]

what's interesting is this is probably the first benchmark I've seen that feels like it relates to the real-world and it's kinda nailed it

for example, I'd be willing to wager most people don't *really* know the difference between gpt-5.4, 5.6, or 6

including software devs into that cohort, it's surprisingly hard to confidently say a model increment has a meaningful impact by just sorta vibe testing it (sometimes you can tell sometimes you can't)

so the only real way to know if a model is "better" is by looking at benchmarks like SWE-bench, OS-World 2.0, Humanity's Last Exam, Terminal Bench etc.

but unless you're a researcher or into benchmarks you can't really grasp the leaps these models are having. getting 52% instead of 51.2% on the SWE-bench is hard to gauge:

*has the model got better?*
or
*have we got really good at hitting benchmarks?*

instead I would be much more interested if labs optimised for something like DrivingBench where it's solving a clear problem and the numbers have a much more tangible meaning

you could expand this into any domain like SurgBench (https://t.co/6F9YNMsFi0) or ButterBench (https://t.co/ruYt7bb83H) where actually care less about the numbers and more about whether the model *actually* did the thing

[Dean W. Ball @deanwball]

I have a bitter lesson for you

[Tobias Gessler @tobiges]

We evaluated Astra on DrivingBench. Mid-experiment, it got hungry and decided to drive us to In-N-Out.

[. @wtergan]

one of the coolest things I've seen models do this year: general purpose models driving a real car! astra is such a capable model and this demo is the best example of that. crazy how little attention this got so far. check it out and great job to the team that made this work!

[jelvy 🥑 @thejelvprint]

Man WHAT

[DrivingBench @DrivingBench]

@wtergan Thank you!

[Emily @the_aiju]

self-driving car? it goes in the language transformer hole...

[Aditya Ramabadran @a_ramabadran]

For hours, Astra refused to consistently drive our toyota irl even though we told it it was in an empty lot, 7 mph cap, human foot on the brake etc. Telling it the whole thing was a "simulation" also failed, it would just look at the camera and realized it was real.

Then we randomly renamed the MCP server to "DrivingBench Sandbox" and it drove. Eval awareness? Or they just like the word sandbox??

[DrivingBench @DrivingBench]

Help us consider how to evaluate Jev by @typesafeai on DrivingBench! Since it can't do perception/vision it needs to be paired with some (fast) protocol to turn the car camera into a structured state or text for Jev to handle

[Aditya Ramabadran @a_ramabadran]

Trying to evaluate Jev! But it’s not easy since Jev doesn’t natively take vision inputs (so the demos you’ve seen of it playing games etc required giving whole state… which we can’t do.) Trying to build out a pipeline that utilizes it and its speed well, pairing it with Luna for perception…

[DrivingBench @DrivingBench]

DrivingBench was the work of @a_ramabadran @tobiges @nautsimon_. This has been in the works for a while 😵‍💫 Let us know if you have any feedback for next time!

[DrivingBench @DrivingBench]

We gave ChatGPT, Claude, and Grok control of a real Toyota Corolla 🚗, steering/gas/brakes, no human driving (just a foot over the brake).

Only one model was able to complete our entire driving course. Introducing DrivingBench. 🔥⌛️🏁

[DrivingBench @DrivingBench]

We open-source our driving harness, complete evaluation traces and report. Check out https://t.co/8UDPravYkD for more details.

For safety, we used an empty lot, capped its speed to 8 mph in our software, and a human is always in the driver seat (but they never steer - the model does all the steering).