Zekai Wang

[Jim Fan @DrJimFan]

The sense of touch is the most criminally under-explored modality in robotics. Imagine doing sleight of hand wearing thick oven mitts. That's exactly how a robot feels today if it were alive. A magnetic piece snapping into place, a paper cup peeling out of a stack, a USB negotiating its way into the port - all invisible to the camera.

Learning how to feel must be a full-stack co-designed effort. We are open-sourcing a principled methodology called "T-Rex":

1. Tactile as first-class citizen of the model. Our mixture-of-transformer runs two clocks asynchronously: a slow visuomotor expert plans the motion, and a fast tactile expert refines it in real time with high-frequency corrections at 4 "touch ticks" per vision tick. Forces change faster than frames arrive, so the architecture had to as well.

2. Open data. The largest tactile dataset ever released to our knowledge: a 50-hour (~5,500 episodes) high-quality, carefully synchronized robot play corpus, collected on SOTA tactile hand hardware with 22 degrees of freedom. Available today on HuggingFace!

3. Training recipe: T-Rex extends our prior work, EgoScale. Human egocentric videos for pretraining, a diverse dose of tactile robot play for mid-training. Our experiments show this bridges contact-free pretraining to contact-rich manipulation remarkably well.

Pixels are cheap and everywhere, but they run out of steam at the moment of contact. Tactile will carry the last mile. The next scaling curve will be measured in hours of touch.

T-Rex is a great collaboration between NVIDIA and Berkeley: 🧵

[David Yin @DavidYin0609]

Please come to our oral and poster presentations on Wednesday — I’ll be there along with @zekaiw04 and @Dantong_Niu to present our work!

Oral: Wednesday, 3:20 — 3:25pm, Room 314
Poster: Wednesday, 3:50 — 4:25pm, Room 314

Paper: https://t.co/VO2ZVwtFj4

[Roei Herzig @roeiherzig]

Have you wondered whether LLMs could directly execute robot tasks?🚨

Excited to share our work, *RoboPrompt*, a framework that enables off-the-shelf text-only LLMs to directly predict robot actions with ICL demonstrations without training! @berkeley_ai

https://t.co/bGLv4oUJ7u

[Zekai Wang @zekaiw04]

Excited to share our new work! It is absolutely shocking to me that LLMs can be used for motion planning through In-Context Learning, without any finetuning!

[Roei Herzig @roeiherzig]

Have you wondered whether LLMs could directly execute robot tasks?🚨

Excited to share our work, *RoboPrompt*, a framework that enables off-the-shelf text-only LLMs to directly predict robot actions with ICL demonstrations without training! @berkeley_ai

https://t.co/bGLv4oUJ7u