Russ Tedrake

Co-Founder and CEO at Walden Robotics. Professor at MIT.

[Russ Tedrake @RussTedrake]

Thanks for hosting me, Brad.

[Brad Porter @BradPorter_]

The best case for wheels I've heard came from the guy who has taught legged locomotion at MIT for 20 years.

AMR safety cases assume you slow proportional to distance and come to rest. A balancing biped may need to take steps to stop. The safety zone explodes and the robot ends up behind a fence.

A robot behind a fence isn't deployed.

Deployed EP12 is live: https://t.co/rqSAc8c9ai

[Russ Tedrake @RussTedrake]

When we first started working on Diffusion Policy, which felt like an "unlock" for fitting human demonstration data, the explanation I gave myself was that "human demonstration data was subtly much more multimodal than we thought, so we need the modeling power of denoising diffusion to fit it properly."

What Michael's paper brings to light is that some of this multimodality was probably exacerbated (more than I realized) by the fact that we've only been giving our models incredibly short observation context lengths.

For me, this is pretty compelling evidence that we should be pushing for policies with long-context lengths and short action prediction horizons.

[Michael Zeng @michzeng_]

Action chunking — especially executing long action sequences open-loop — is widely used in imitation learning for robotic manipulation. Why is it so effective and do we really need it? We find a key reason:

Long open-loop execution helps short-context policies imitate non-Markovian experts.

With this insight, we show how to move beyond open-loop execution: extending policy context restores reactivity while achieving even higher task performance.

🧵(1/5)

[Walden Robotics @WaldenRobotics]

Building great robots takes great partners.
Proud to be collaborating with @SamsungSDSA on a proof of concept focused on manufacturing use cases, combining Walden's robotics platform with Samsung SDS' manufacturing expertise.
More: https://t.co/G9OIaazh8C

[Russ Tedrake @RussTedrake]

Thanks to @HeySamantha and @Bloomberg for covering our launch https://t.co/KlT95dRp6s

[Russ Tedrake @RussTedrake]

https://t.co/PkNTPKvOrT

[Russ Tedrake @RussTedrake]

In January, I started "building something new" with an incredible team. Today I finally get to share some first details about what we've been building. We've called it Walden Robotics (https://t.co/LVSojsX8aJ).

I thought long and hard about my own reasons for starting this company. It's not only about the robots. It's also about people. I've tried to capture those thoughts in my first Walden blog post: https://t.co/Lmo1CU8tLF

It's been an incredible ride so far. Within just a few months of forming the company, we were already operating a general-purpose robot with an end-to-end policy in production in one of the most important factories in North America. It's amazing at how much I've already learned from that experience. There is a lot of work to do, but the mission has never been so clear.

Please help me welcome Walden Robotics into the world. And stay tuned for more updates!

https://t.co/vaVRt0JFY1

[Russ Tedrake @RussTedrake]

When it comes to diffusion policies, VLAs, and LBMs, we all know that data quality matters. Many of us have spent considerable time filtering data, and discarding large amounts of data that might not be considered to be sufficiently high quality. This feels very inefficient.

We’ve seen several proposals of late for how we might learn from suboptimal data. pi0.7, for instance, suggested that we learn models conditioned on the data quality.

Adam’s new paper exposes a different (complementary) angle. When we’re training denoising diffusion models, each data point is potentially used several times — at different noise levels. **The value of a data point might be different depending on the noise level.**

Adam’s work explores this nicely and shows that he can extract value even out of significant portions of the OXE dataset that many of us had previously discarded as suboptimal.

Definitely worth having a look.

[Adam Wei @adamwei_]

🤖 We introduce Ambient Diffusion Policy, a simple and principled method for training policies with suboptimal data in robotics.

Suboptimal data is everywhere in robotics…
❌ Data filtering is wasteful
❌ Co-training learns both good and bad features
✅ Ambient Diffusion Policy selectively learns useful features via noise-dependent data usage

👇🧵(1/5)

[Russ Tedrake @RussTedrake]

@chetan_ Just to be clear, we can and are doing significantly more than pick and place with the scenes/assets created here. That's always been an emphasis for me.

[Russ Tedrake @RussTedrake]

I've been saying for years that the biggest challenge for simulation in robotics is not actually the physics engine (although you do have to get that right). The real challenge is capturing the *diversity* of the real world. There was no doubt that generative AI had the potential to change that, but it's still amazing to see it take shape.

Watching Nick's incredibly fast progress has convinced me that content generation might not actually be a bottleneck anymore. This is a beautiful combination of hardened tools for e.g. low-level mesh processing with the latest tools for generative asset creation, wrapped in a powerful agentic workflow. Please do give it a try and share your feedback.

[Nicholas Pfaff @NicholasEPfaff]

Meet SceneSmith: An agentic system that generates entire simulation-ready environments from a single text prompt.

VLM agents collaborate to build scenes with dozens of objects per room, articulated furniture, and full physics properties.

We believe environment generation is no longer the bottleneck for scalable robot training and evaluation in simulation.

Website: https://t.co/UZklSkJe9V

👇🧵(1/8)

[Russ Tedrake @RussTedrake]

@JeremySMorgan3 We're working on and eagerly watching all of the progress in real2sim. I do think capturing particular scenes (e.g. the robot failed at a task in the field, and you want to reconstruct that in sim) is going to be very high value when we really get it right.

[Russ Tedrake @RussTedrake]

It's a good question. We have articulated objects.. typically the only actuated object in the scene is the robot, which we insert manually (and you want to connect that to your control stack, etc, too).

We've found that the basic friction parameters are pretty good, but I agree that this could be a nice way to further increase the fidelity.

[Russ Tedrake @RussTedrake]

RL training in sim has been incredibly powerful for locomotion, because it turned out that training on a handful of terrains was sufficient to generalize to the real world.

We haven't seen the same for manipulation. Open-world manipulation has, instead, been leaning on real world data collection (and imitation learning). The main reason for this, I would argue, is because we haven't captured the diversity of the manipulation problem in our simulators yet.

It's still possible that we could find a minimal set of scenarios that capture the full complexity of the dexterity of manipulation, but manipulation also requires world knowledge (e.g. pick up the mugs from the handles, etc) so you have to get that from somewhere too -- strong multimodal base models and/or imitation. Assets + physics engines are not enough.

[Russ Tedrake @RussTedrake]

My personal journey on this has been in Drake (https://t.co/xIwJ1HAbJD).

First step: The assets have to be good. Garbage in, garbage out. (People under estimate how low quality most assets are... even the models you get straight from robot manufacturers.)

Second step: Get the physics engine right. We've been pounding on getting better contact models (e.g. Drake's hydroelastic contact model) and stronger and stronger numerical methods for contact simulation in Drake. Alejandro's papers are a great summary of that work: https://t.co/dTgiqS2wrD

[Boyuan Chen @BoyuanChen0]

Introducing Large Video Planner (LVP-14B) — a robot foundation model that actually generalizes. LVP is built on video gen, not VLA. As my final work at @MIT, LVP has all its eval tasks proposed by third parties as a maximum stress test, but it excels!🤗
https://t.co/wjD54YFK3k

[The Humanoid Hub @TheHumanoidHub]

Boston Dynamics and Toyota Research Institute are advancing humanoid robotics with Large Behavior Models (LBMs) for Atlas.

[Russ Tedrake @RussTedrake]

@Anweshac1211 yes. Of course the distribution and quality of the data matters.

[Russ Tedrake @RussTedrake]

TRI's latest Large Behavior Model (LBM) paper landed on arxiv last night! Check out our project website: https://t.co/AV2cmfeX40

One of our main goals for this paper was to put out a very careful and thorough study on the topic to help people understand the state of the technology, and to share a lot of details for how we're achieving it.

https://t.co/EVFLJAY6Zu

[Russ Tedrake @RussTedrake]

This was a massive effort by the entire team, with a number of individuals really pouring their hearts into this paper. The paper is packed full of (too many?) details. Your comments and feedback would be very welcome.

[Russ Tedrake @RussTedrake]

Probably my favorite plot from the paper, which sums it all up, is this one.
The plot compares performance using different amounts of pretraining data used before training a new task: 0% (aka single task), 25, 50, or 100% of TRI’s data, then 100% of TRI’s data + all of the open-source robot data (the red line) that we’ve curated. It’s just awesome that the distributions over task completion are so tight and that trends as we increase data are so consistent. The results show clearly that with pretraining, we can train a novel skill with substantially less data or use the same amount of data and get much better task performance. And the benefits appear to continue with more data.