Leo Dong

Industrial automation @rhoda_ai
Prev @Stanford

[Leo Dong @leooeld]

@vincesitzmann @DaveLindell Thank you! It was awesome working among the extended @GordonWetzstein Cinematic Universe 😂

[Mike Schroepfer @schrep]

This is rad!

[Vinod Khosla @vkhosla]

Robots making rapid progress in "learning to learn"

[Rhoda AI @RhodaAI]

What can a robot do without vision?

We gave our robot a sense of touch and showed it can recognize and retrieve objects with its cameras blindfolded.

Introducing Rhoda’s new tactile sensor—it feels pressure, texture, vibration, slip, and temperature, just like your fingertips.

[Leo Dong @leooeld]

Video from: https://t.co/C5qkcdlFIP

[Leo Dong @leooeld]

If your robot does this, cut it some slack! Humans do it too when our fingers are anesthetized and lose our sense of touch.

That's why we are bringing tactile sensing to our Direct Video-Action model at Rhoda. Read more in our blog post: https://t.co/JK59RQHtOa

[Vincent Sitzmann @vincesitzmann]

Touch is essential to human manipulation. Yet, we have so far not had a tactile sensor that can do what human skin can: sense texture, firmness, vibration at a viable form factor. At Rhoda, we have built a new sensor in-house that can do all of these things, while being much much smaller than commercially available ones!

[Qianqian Wang @QianqianWang5]

Such a cool demo of tactile sensing: recognizing and retrieving objects by touch alone! Congratulations to the Rhoda team!

[Leo Dong @leooeld]

@AndrewEJXie @RhodaAI @DaveLindell Amazing work Andrew! So glad to have you with us in the summer!

[Joanne Truong @joannetruong]

Exciting to see how naturally DVA extends to new modalities! Most robot policies still learn without touch, even though it matters most when vision falls short: occlusion, contact, etc.

It was so much fun to see this come together, congrats @leooeld @AndrewEJXie @DaveLindell!

[Leo Dong @leooeld]

When the project first started, I knew nothing about tactile sensing, and this video @DaveLindell shared really opened my eyes to just how crucial touch is to human manipulation. If you anethesize your fingers and take away touch, you can't even pick up everyday objects! https://t.co/EG1vVxouCx

[David Lindell @DaveLindell]

We gave our robots a sense of touch!

Excited to finally share this work with @AndrewEJXie, @leooeld, and the @rhoda_ai team!

Some technical details on the tactile sensor and robot policy: 🧵

[Leo Dong @leooeld]

@DaveLindell @AndrewEJXie It was a lot fun to work with you and @AndrewEJXie and I learned so much!

[Leo Dong @leooeld]

(5/n) As humans, our sense of touch complements vision and is essential in its own right, allowing us to manipulate objects even when we can’t see them clearly.

Similarly, our robots can combine vision and touch for contact-rich industrial manipulation, where poor lighting or occlusion can impair vision.

Our DVA model can now imagine what the robot will see—and what it will feel. Read more: https://t.co/UaKhBaX1i2

[Leo Dong @leooeld]

(4/n) In the DVA paradigm, learning to predict the next tactile frame is thus equivalent to learning a robot policy that uses tactile signals to guide its actions. We test how much the policy actually uses tactile signals (instead of ignoring them) with an extreme challenge: we ask the robot to perform a manipulation task entirely without vision. With its cameras blindfolded, the robot has to feel around inside a box, search for objects, and pick out only the textured balls from a mix of balls.

[Leo Dong @leooeld]

(3/n) The connection to touch is in the representation. Contact measurements form spatial maps that change over time, which we can think of as a “tactile video.” DVA’s web-scale pre-training gives it a foundation for modeling extremely diverse web videos, making it easy to learn to predict tactile videos through post-training.

[Leo Dong @leooeld]

(1/n) Watch our robot search a box and find textured balls, entirely without vision!

Over the past few months, it’s been incredibly fun to work on bringing touch into our Direct Video-Action (DVA) model. Here’s how our video model is “seeing” through touch 🧵

[Rhoda AI @RhodaAI]

What can a robot do without vision?

We gave our robot a sense of touch and showed it can recognize and retrieve objects with its cameras blindfolded.

Introducing Rhoda’s new tactile sensor—it feels pressure, texture, vibration, slip, and temperature, just like your fingertips.

[Leo Dong @leooeld]

(2/n) DVA uses a causal video model as a robot policy. It is a next-frame generator that imagines what the immediate future should look like. An inverse dynamics model realizes that imagined future by translating into actions. The robot moves, observes the result, and the video model predicts again.

[Leo Dong @leooeld]

@RhodaAI Today our Direct Video-Action model learns a new modality, and it's "seeing" through touch!