Can we achieve state-of-the-art for mainstream E2E and vectorized driving benchmarks without using any human demonstration data?
Terra series at @AppliedInt now tops the NAVSIM, HUGSim, nuPlan val14 and InterPlan benchmarks, powered by self-play RL without imitation, scaling in closed loop with all agents being reactive.
🚘 With TerraZero released a few months ago, we set the SOTA on vectorized benchmarks like nuPlan val14 and InterPlan, powered by our super high-throughput self-play RL framework experiencing 25 centuries of challenging driving scenarios.
📽️ In June this year, TerraTransfer was released as the first of its kind to bring the SOTA vectorized policy from self play (TerraZero) to an E2E policy via image-to-vector encoder alignment, which topped the HUGSim closed-loop score.
🔚 This week, we are thrilled to release TerraTransfer++, a demonstration-free E2E planner setting the SOTA for NAVSIM and HUGSim, by adding planning capability from self-play rollout experience to TerraTransfer.
Setting SOTA for public benchmarks is just a beginning, as we have seen the journey from pioneering work like UniAD to successful mass production of E2E ADAS in the industry, but what makes me quite excited is to see the strong signal from a new paradigm with potential to refactorize how the next generation autonomy can be developed, and bring the scalability, safety and performance to the next level.
We are the believer of data and scaling, but not just human demonstration data and open-loop scaling like imitation or open-loop RL (AV2.0) - we have the bet on closed-loop scaling with both synthetic and real-world data (AV3.0), and the Terra series is the starting point of the journey towards real product with our bet.
Great work from the team @caffe_tt, Weixin Li, Zhouchonghao Wu, @AkshayRangesh, Saarth Bonde, Grantland Hall, @ChenTangMark, Yihan Hu, for creating TerraTransfer and TerraTransfer++!
TerraTransfer++ report: https://t.co/2cEGKqAXyN
TerraTransfer++ website: https://t.co/K4zEo7k4jN
TerraTransfer report: https://t.co/kMmmYF0gaS
At this year’s Agentic AI Summit hosted by @BerkeleyRDI, our Chief Scientist, @Wei_ZHAN_ shared some of the cutting edge research he has been leading in the physical AI space.
As the industry races to deploy L2++ ADAS with imitation-learning-based E2E, Wei tackled a key question: what if end-to-end autonomy could be trained without imitation at all, relying only on reinforcement learning? An idea that runs against the current status quo.
Here are some of his insights
🧵
To obtain robust autonomy, it has always been the desire for planners to experience large-scale, high-value scenarios in ultrafast closed-loop training with minimal unit cost of data and compute, ideally self-evolving even without offline human demonstration.
TerraZero is our straightforward response at @AppliedInt to such a desire by scaling self-play reinforcement learning with procedural simulation and zero demonstration:
- Insanely fast in closed loop, i.e., 2.8 million simulation steps per second on 8 GPUs;
- Endlessly streaming hard and diversified corner cases with a procedural generation;
- Robust self-play recipe trading sample efficiency for compute efficiency;
- Zero demonstration in training, and zero-shot emergence across domains (cities, datasets, etc.);
- Minimal unit cost for experiencing high-value data points with a highly scalable framework.
So excited to see that one stack powered by TerraZero can serve various outcomes:
- Setting the state-of-the-art in the challenging, long-tail planning benchmark (InterPlan) with a clear edge;
- Topping the realism among demonstration-free methods in sim agent benchmark (WOSAC);
- Comprehensive practicality with policy covering heterogeneous agents (VRUs, trucks, trailers, etc.) and tackling noisy policy inputs.
On our website, you can also select arbitrary combinations of scenario/agent/view features and see how TerraZero works - play with it!
Really impressed by the potential of large-scale, high-value synthetic data in closed loop powered by ultrafast training, and TerraZero is just the beginning of our Terra Series research - committed to pushing the boundaries of physical AI learned in closed loop with scaled simulation and minimal demonstration. More follow-up work will be announced soon, stay tuned!
TerraZero report: https://t.co/iJA0NdHkF8
TerraZero website: https://t.co/aaqJpNCduM
Terra Series website: https://t.co/5s4CiOoBd5
Today, we’re officially announcing our Self-Driving System for Automotive — our L2++ ADAS product for passenger vehicles. 🚀
Meet the system under the hood: https://t.co/PG8AxbOwGF
#autonomousvehicles#vehicleintelligence#adas
Applied Intuition is collaborating with @OpenAI to advance next-generation, AI-powered experiences in vehicles worldwide.
Learn more about this collaboration: https://t.co/7tzIi1CXfU
#AI#vehicleintelligence#OpenAI
Crazy AF. Paper studies @_akhaliq and @arankomatsuzaki paper tweets and finds those papers get 2-3x higher citation counts than control.
They are now influencers 😄 Whether you like it or not, the TikTokification of academia is here!
https://t.co/gXCvUBrMMY
BREAKING: San Francisco greenlights all-day paid self-driving robotaxis, marking a historic step for Cruise and Waymo. Despite concerns, California's Public Utilities Commission approves their expansion. https://t.co/vEGi5oV80h
OpenAI's new ChatGPT explains the worst-case time complexity of the bubble sort algorithm, with Python code examples, in the style of a fast-talkin' wise guy from a 1940's gangster movie:
Today @Cruise launched *daytime* driverless rides. Service is live for Cruisers and will launch to the public soon.
First ride was this morning around 8:30 am:
We spent weeks crunching numbers, talking with experts, writing crazy code, drawing complex diagrams to explore the effectiveness of travel restrictions. Here's why it hasn't been enough to stop the spread of the #coronavirus. #COVID19 https://t.co/VXniVtKUQA
3D BAT: A Semi-Automatic, Web-based 3D Annotation Toolbox for Full-Surround, Multi-Modal Data Streams
w/ @AkshayRangesh
Compares open and commercial annotation tools and open sources a new one
ArXiv
https://t.co/DoTbVK4D3Z
Github
https://t.co/XtXjvuVm8Z
The USA Constitution, with each Amendment as a git commit.
I can't imagine a better tool to understand the historical evolution of this document…
https://t.co/SoptlooyNO
We've observed AIs discovering complex tool use while competing in a simple game of hide-and-seek. They develop a series of six distinct strategies and counterstrategies, ultimately using tools in the environment to break our simulated physics: https://t.co/2lbOxo19rL