Yevgen Chebotar

Robotic foundation models @NVIDIA 🤖
Previously @GoogleDeepMind (VLAs, RT-2, Offline RL) and @Figure_robot (Helix models)

[Yunfan Jiang @YunfanJiang]

Exactly! Tool use adds another layer of dexterity and sequencing. Here's an uncut end-to-end policy using a drill to assemble a toy car. Notice how the robot makes micro-adjustments to the drilling angle and still completes the fastening even when the object isn't perfectly aligned. That's the kind of robustness we're excited about. 🦾🔩

[Yevgen Chebotar @YevgenChebotar]

Long visuomotor context is one of THE most important missing parts for robotic foundation models, there is no way around it if we want generic policies that remember more than a couple of camera frames. However, adding visual context naively significantly increases inference time on the robot and doesn’t scale beyond very short horizons.

In RoboTTT, we found a way to scale it to orders of magnitude more to up to 8k timesteps without a significant increase of inference time through a principled integration of Test-Time-Training (TTT) into the foundation models. The context is saved in fast weights of the model that can be updated at inference in constant time.

This opens a range of new applications and possibilities:
- Performance on long-horizon dexterous tasks keeps on increasing with context length. It doesn’t require any separate planner for coordinating the subtasks, can self-correct and becomes robust to perturbations.
- One-shot imitation from human videos can be done in-context by just concatenating human videos and robot trajectories.
- We can develop novel in-context improvement algorithms, such as DAgger distillation that meta-learns to map suboptimal robot actions to human corrections within the same sequence.

Check out the deep dive and the paper for more details!
https://t.co/HsH1cUwObl

[Yunfan Jiang @YunfanJiang]

We scaled robot policies to 8K timesteps of visuomotor context, orders of magnitude beyond current SoTAs, at constant inference latency.

Introducing RoboTTT 🤖

With minutes of experience in context, our robots:
🎥 one-shot imitate human video demos
📈 improve themselves during deployment
🛡️ recover from perturbations
⚙️ complete a 5-minute, 10-stage assembly end to end
🌐 https://t.co/4OoK0e9tKa

Dive in 🧵

[Ruijie Zheng @ruijie_zheng12]

Proud to introduce EgoScale: We pretrained a GR00T VLA model on 20K+ hours of egocentric human video and discovered that robot dexterity can be scaled, not with more robots, but with more human data. A thread on 🧵what we learned. 👇

[Zhengyi “Zen” Luo @zhengyiluo]

SONIC is now open-source!

Generalist whole-body teleoperation for EVERYONE!

Our team has long been building comprehensive pipelines for whole-body control, kinematic planner, and teleoperation, and they will all be shared.

This will be a continuous update; inference code + model already there, training code and gr00t integration coming soon!

Code: https://t.co/7u3SBxzXU9
Docs: https://t.co/HpDLkTCSMF
Site: https://t.co/D3i4KlnLLr

[Yevgen Chebotar @YevgenChebotar]

We are moving towards world-model based backbones for robotic policies, pre-trained on web-scale videos and outputting robotic actions directly within the same diffusion model!

There are new levels of transfer to unseen tasks and motions being unlocked, something that has been missing in VLM-based pre-training ever since the early days of RT-2 VLA models, which although generalizing well to new objects and semantics always struggled with new “verbs” and “motions”. We also observe signs of efficient cross-embodiment transfer to new robots with a little amount of data.

There are a lot of optimizations that allow us to run a 14B world action model in real time! My favorite trick is DreamZero-Flash, which while still denoising videos and actions jointly, introduces separate noise schedules, such that actions can be denoised much faster than videos to enable higher frequency control!

Check out the website, paper and especially the eval gallery!
https://t.co/btABHbiiyH

[Joel Jang @jang_yoel]

Introducing DreamZero 🤖🌎 from @nvidia

> A 14B “World Action Model” that achieves zero-shot generalization to unseen tasks & few-shot adaptation to new robots
> The key? Jointly predicting video & actions in the same diffusion forward pass

Project Page: https://t.co/qhygDzu6NY
🧵 (1/10)

[Yevgen Chebotar @YevgenChebotar]

Excited to join the NVIDIA GEAR team to help build the next generation of open robotic foundation models!

[Yevgen Chebotar @YevgenChebotar]

We've made great progress on Vision-Language-Action Models for humanoids in our new Helix model!

Check out the technical report for more details:
https://t.co/muMeO2Log7

[Figure @Figure_robot]

Meet Helix, our in-house AI that reasons like a human

Robotics won't get to the home without a step change in capabilities

Our robots can now handle virtually any household item:

[Yevgen Chebotar @YevgenChebotar]

The path to VLAs lies through VLMs. A very nice intro for everyone interested in working with Vision-Language Models:
https://t.co/pRpSebf3Tb

[Yevgen Chebotar @YevgenChebotar]

Congrats everyone, 170+ authors and contributors, great to see the robotic field coming together!

[Karl Pertsch @KarlPertsch]

Our OpenX paper won best paper at ICRA! Congrats to all my co-authors! 🎉🎉
This is an ongoing effort, we recently added new datasets from the community that double the size of the OpenX dataset -- keep 'em coming! :)

Check datasets & how to contribute: https://t.co/gX5Ir30s5B

[Yevgen Chebotar @YevgenChebotar]

Some personal updates!
Excited to join the team @Figure_robot to help building AI for the robot age! 🤖

[Yevgen Chebotar @YevgenChebotar]

Turns out classification loss works surprisingly well for value-based RL, also some nice gains when used with Q-Transfomer!

[Aviral Kumar @aviral_kumar2]

Super simple code change to get value-based deep RL scale *much* better w/ big models across the board on Atari games, robotic manipulation w/ transformers, LLM + text games, & even Chess!

Just use classification loss (i.e., cross entropy), not MSE!!

https://t.co/0IHSgN4pBj🧵⬇️

[Yevgen Chebotar @YevgenChebotar]

RT-H learns a hierarchy all the way from high-level tasks through low-level “language motions” to robot actions!

✅ Improved performance and generalization through better data sharing
✅ Automated grounded “bottom-up” labeling
✅ Ability to intervene and correct with language

[Suneel Belkhale @suneel_belkhale]

Is language capable of representing low-level *motions* of a robot?

RT-Hierarchy learns an action hierarchy using motions described in language, like “move arm forward” or “close gripper” to improve policy learning.

📜: https://t.co/61KZfQNpkY
🏠: https://t.co/voIvzq5Mek (1/10)

[Ted Xiao @xiao_ted]

Had a great time today with @YevgenChebotar and @QuanVng visiting @USCViterbi to give a talk on “Robot Learning in the Era of Foundation Models”. Slides out soon, packed with works from *just the past 5 months* 🤯

Thanks to @daniel_t_seita for hosting!

[Yevgen Chebotar @YevgenChebotar]

Presenting RT-2 poster at CoRL!

https://t.co/fA39xewX5O

[Quan Vuong @QuanVng]

Pictures taken at RT-2 poster at @DannyDriess requests ; )

@YevgenChebotar

We miss you @TianheYu

CC @hausman_k

[USC Thomas Lord Department of Computer Science @CSatUSC]

How many @CSatUSC grads does it take to create a breakthrough robot? Three, apparently!

Congrats @YevgenChebotar @hausman_k and @ryancjulian who worked on Google DeepMind's revolutionary RT-2 AI model.

Find out more ⤵️
https://t.co/D4HqA7d7zH

@gauravsukhatme @USCViterbi

[Yevgen Chebotar @YevgenChebotar]

@_ntr_p_ There was a technical problem with the old website address, the updated website is at
https://t.co/x6L5qBZutA

[Yevgen Chebotar @YevgenChebotar]

Offline RL strikes back! In our new Q-Transformer paper, we introduce a scalable framework for offline reinforcement learning using Transformers and autoregressive Q-Learning to learn from mixed-quality datasets!

Website and paper: https://t.co/SntNYC9Pk3 🧵

[Yevgen Chebotar @YevgenChebotar]

Exciting times for Robot Learning! 60 datasets from 22 different robots and 21 institutions combined in a single Open-X Embodiment data repository, resulting in over 1 million episodes and improved RT-X models! Amazing and a very important collaboration across the world! 🤖🌐

[Quan Vuong @QuanVng]

RT-X: generalist AI models lead to 50% improvement over RT-1 and 3x improvement over RT-2, our previous best models. 🔥🥳🧵

Project website: https://t.co/GAlvFdqwx5

[Yevgen Chebotar @YevgenChebotar]

Joint work with @QuanVng, @AlexIrpan, @hausman_k, @xf1280, @Yao__Lu, @aviral_kumar2, @TianheYu, @AlexHerzog001, @KarlPertsch, @keerthanpg, @julianibarz, @ofirnachum, @Kanishka_Rao, @chelseabfinn, @svlevine

[Yevgen Chebotar @YevgenChebotar]

Our real robot policies significantly improve upon RT-1 and other baselines when trained on limited amount of human demonstrations by leveraging autonomously collected negatives and dynamic programming properties of Q-learning.