Eric Chan

Chief Scientist of Rhoda AI

prev. PhD Student, Stanford University

[Eric Chan @ericryanchan]

Today, we announce our team’s progress in pursuing a different type of foundation model for robotics: the Direct Video Action Model (DVA), which does our best to take robotics and turn it into a generative modeling problem we can scale.
Technical blog: https://t.co/GMsxnC5wbJ

[Vincent Sitzmann @vincesitzmann]

Thrilled to introduce Mirror, our first "learning-from-experience" project! Assume you had a video generative model that, no matter your situation, could always generate video of what it would look like to solve a task. How can you put that plan into action? Clearly, you need to learn some video-to-action mapping - but where do you get the data for that? It turns out you can collect that data in an amazingly sample-efficient way by *incrementally following the video planner*! (1/n)

[Eric Chan @ericryanchan]

The result is a new way of learning representations that is stable, informed by the theory, and most importantly, scalable. And it's a general recipe, so maybe we'll start to see some of these techniques applied to new domains and modalities!

[Eric Chan @ericryanchan]

Excited to share @BaadeAlan's latest work: a new approach to representation learning designed specifically for generative modeling!

Semantic representations like DINO have become remarkably effective tokenizers for image and video generation. But DINO was never designed for this task. Its success in generative modeling is almost accidental. What if we learned representations specifically to make generation easier?

[Alan Baade @BaadeAlan]

1/ The tokenizer is a new scaling axis!

We introduce an approach for diffusable representation learning that's principled, scalable, and works when DINO/LeJEPA collapse.

We get joint tokenizer+diffusion scaling laws and it's compute-optimal vs raw diffusion

w/ @drfeifei @GordonWetzstein @jcjohnss @ericryanchan @vincesitzmann @KyleSargentAI

Thread, Paper, and Blogpost below

[Eric Chan @ericryanchan]

One key insight is a connection to contrastive learning. Contrastive methods learn to distinguish pairs of observations that occur together from randomly paired observations. The optimal score for this task is pointwise mutual information (PMI): a measure of how much more likely two observations are to occur together than by chance.

Rather than learning PMI indirectly through contrastive learning, our method uses likelihoods from a generative model to estimate PMI and directly supervise the representation!

[Vincent Sitzmann @vincesitzmann]

Very excited about this one: a completely new representation learning objective, with a proper theory of what it converges to: the representation of an image converges to features such that their dot product with another image's representation is their pointwise mutual information! @BaadeAlan knocked it out of the park :) (1/n)

[Leo Dong @leooeld]

(1/n) Watch our robot search a box and find textured balls, entirely without vision!

Over the past few months, it’s been incredibly fun to work on bringing touch into our Direct Video-Action (DVA) model. Here’s how our video model is “seeing” through touch 🧵

[Vincent Sitzmann @vincesitzmann]

Touch is essential to human manipulation. Yet, we have so far not had a tactile sensor that can do what human skin can: sense texture, firmness, vibration at a viable form factor. At Rhoda, we have built a new sensor in-house that can do all of these things, while being much much smaller than commercially available ones!

[Eric Chan @ericryanchan]

I often get asked how world models will integrate tactile data, when they’re trained primary from video.

Here’s a simple solution that works surprisingly well: embed the tactile information into the video stream, as pixels. At Rhoda, we’ve developed inexpensive and sensitive tactile sensors, and have demonstrated their capabilities, including allowing a blind robot to sense. It’s an exciting move towards greater dexterity!

[Rhoda AI @RhodaAI]

What can a robot do without vision?

We gave our robot a sense of touch and showed it can recognize and retrieve objects with its cameras blindfolded.

Introducing Rhoda’s new tactile sensor—it feels pressure, texture, vibration, slip, and temperature, just like your fingertips.

[Rhoda AI @RhodaAI]

Continuing our RhodaAtWork series with another look at Direct Video-Action (DVA) in action.

This time, a single model handles multiple tasks across two home appliances: refrigerator EPS foam installation and washing machine drain hose routing.

[Rhoda AI @RhodaAI]

More industries. More real customer work.

Introducing RhodaAtWork, a series on the customer workflows our robots are learning to handle across manufacturing and logistics.

First up: unpacking, scanning and stacking component reels for electronics manufacturing.

[Stephen James @stepjamUK]

Rhoda AI checked whether scaling web video pretraining actually helps real robots, using a real customer task. Unpacking 10kg boxes of bearings and sorting the packaging.

Turns out yes. Bigger models do better, more pretraining compute does better, and the compute gap is widest when you don't have many robot demos.

Refreshing to see a company display this amount of rigour. Rare these days.

[Eric Chan @ericryanchan]

We recently came out with our blog post, Does Scaling Web-Video Pre-training Help Real Robots Do Real Work?

This is a pretty exciting moment for robotics, because as far as I am aware, it is the first evidence that robot foundation models can scale on internet video data; not merely on teleoperation, UMI data, or egocentric data, but on the data you can truly find anywhere.

It’s the necessary requirement for reaching massively powerful models.

[Rhoda AI @RhodaAI]

At Rhoda, we care deeply about the science of pre-training for robotics.

In one of the most rigorous studies of its kind, over thousands of trials and hundreds of hours of robot evaluations, we show how scaling web-video pre-training leads to better real-world robot performance.

[Eric Chan @ericryanchan]

An insightful study from @tongzhou_mu and the @RhodaAI team that examines how web video pre-training translates to robot performance.

The gist: pre-training doesn’t just help; it scales, i.e. you can continue to reap greater benefits as you put more effort into pre-training. A better pre-trained model will lead to a more robust policy.

But how you achieve that pre-training quality is up to you! Whether by scaling the model, or scaling your training tokens, it’s a design choice informed by your compute constraints, training budget, and how much you care about compute optimality.

[Tongzhou Mu 🤖🦾🦿 @tongzhou_mu]

Does scaling pre-training on general web video improve a complex manipulation task in real deployment?
We scale model size and pre-training compute, and test on one industrial task.
Yes. The better a pre-trained model predicts web video, the better its post-trained policy. 🧵

[Vincent Sitzmann @vincesitzmann]

I was a guest on the @MIT_CSAIL podcast to discuss robots, AI, and what the near future might look like. @klgiven did a great job steering the conversation, and I think much of our chat is accessible to non-experts - hope you find it interesting!
https://t.co/RLsAxGoyG1

Also on Spotify:
https://t.co/Fx2QHsVWja

[Alan Baade @BaadeAlan]

I'll be at ICML to present Latent Forcing!

Come stop by the poster on Tuesday at 2:00 PM KST, Hall A #3507

Reach out if you want to chat about the future of generative models!

[Lester Li @sizhe_lester_li]

Robot learning is moving beyond policies built for one robot, one scene, one task.

At MIT, we’re exploring a different path: turning video world models into embodiment-agnostic robot policies.

Introducing VERA: a 14B video-to-action system that controls robots across embodiments, skills, and environments.

From zero-shot pick-and-place on a real Panda arm to contact-rich cube reorientation with a 16-DoF robotic hand.

Different robots. Different environments. Different tasks.
Same video planner. Same weights.

We’re open-sourcing everything so you can fine-tune VERA for your own robot setup too. Deep dive in the thread:

🔗 https://t.co/hzuYZ2m5lS
🧵 (1/7)

[Eric Chan @ericryanchan]

Delighted to have you on the team! Here’s to many new insights together :)

[Vincent Sitzmann @vincesitzmann]

I am thrilled to join https://t.co/gwO0u7PhYH @RhodaAI as an advisor, where I am helping harness the abilities of large-scale pre-training and video models for robotics, putting many of my lab's research learnings of the past few years into practice! I will be in-person at the Mountain View office for part of the summer - reach out if you want to chat :)

[Vincent Sitzmann @vincesitzmann]

Introducing MilliVid, our new method for long-context video generation! MilliVid creates videos that are consistent over long time spans, without using retrieval heuristics or 3D maps! (1/n)
https://t.co/evmf5dL5Sg

[Qianqian Wang @QianqianWang5]

The shell game is a fun challenge that cannot be solved by looking at a single frame. The model has to track every move, from the moment the object is hidden. Excited to share this!

[Rhoda AI @RhodaAI]

Here’s something we’ve never seen done before.

Real-world tasks are long and ambiguous. Solving them requires visual memory and state tracking. Most robot policies only see the last few frames. Ours doesn't.

We put our DVA, FutureVision, to the perfect testbed: the shell game 🐚. The DVA nails it.