Jaden Clark

PhD @Stanford @KnightHennessy. AI, robotics, and conservation

[Jaden Clark @jadenvclark]

Can we enable robots to develop a sense of touch without forgetting what they learned from large-scale vision-only pretraining?

Introducing MultiSensory World Model (MuSe) 🌍: A new approach for finetuning visuomotor policies on minimal data from new sensor modalities, such as force/torque (F/T)

With Muse, touch learned later improves skills learned earlier — a small amount of F/T data on new tasks improves zero-shot on diverse pretraining tasks that were never supervised with F/T

We believe MuSe provides a practical pathway towards training multisensory foundation models that leverage both abundant vision data, and smaller multisensory datasets 🧵👇

[Jaden Clark @jadenvclark]

We've been using iPhumi for MuSE/UMI-FT - iPhone is a big upgrade from original UMI design with better slam and easier UI!

[Austin Patel @austinapatel]

We are also excited to release iPhUMI ("eye-foo-me”)! It solves the localization challenges with the GoPro UMI, enabling rapid data collection across diverse environments and tasks. During deployment, iPhUMI lets you command your robot via demonstration.

https://t.co/HYifek0RmS (7/8)

[Yifan Hou @YifanHou2]

Tactile/force data are critical but rare. They can never reach the scale of pretraining tasks, so we got to find intelligence in other ways.

In MuSe, we show that finetuning with a small amount of force data can even improve pretrain tasks, given the model the ability to do force prediction on tasks with no force data before.

The ability is enabled by three key modeling designs. Checkout Jaden's post for details!

[Shuran Song @SongShuran]

I'm increasingly interested in the problem of *multisensory continual learning*, since it feels inevitable for robotics.

Unlike vision, many robot sensors (e.g., force/torque, tactile, audio) are highly task- and system- specific. It's unrealistic to expect a single pretraining dataset to contain every future sensor. And as robotics evolves, we'll keep building new sensors.

So the question is: Can we plug a new sensor into a pretrained vision-only foundation model without forgetting everything it already knows?

Better yet, can the new sensor actually improve the model's existing vision-based skills?

That's exactly the question that motivated MuSe 👇

[Changhao Wang @ChanghaoWang78]

MuSe is our first attempt to enable robots to learn touch while preserving the generalization gained from large-scale vision pretraining. More surprisingly, touch learned later can in turn improve the skills acquired earlier during pretraining.

[Mandi Zhao @ZhaoMandi]

WAMs + force data improves on both pre-training and fine-tuning tasks:

[Jaden Clark @jadenvclark]

This was a really exciting collaboration with @ChanghaoWang78 @YihuaiGao Seongheon Hong @Hojung_Choi_ Mark Cutkosky @YifanHou2 and of course @SongShuran

Website: https://t.co/bVDA0Y789q
Paper: https://t.co/wdq71oyGYc

[Jaden Clark @jadenvclark]

MuSe leverages broad pretraining data collected with UMI grippers to improve generalization on contact-rich tasks from finetuning (forward transfer)

[Jaden Clark @jadenvclark]

While naive finetuning leads to catastrophic forgetting, MuSe exhibits backward transfer: improving in pretraining tasks that never seen F/T conditioning in training

[Jaden Clark @jadenvclark]

MuSe outputs force/torque predictions enabling policy to adapt its compliance for forceful yet precise contact rich interaction 💪

[Jaden Clark @jadenvclark]

Can we enable robots to develop a sense of touch without forgetting what they learned from large-scale vision-only pretraining?

Introducing MultiSensory World Model (MuSe) 🌍: A new approach for finetuning visuomotor policies on minimal data from new sensor modalities, such as force/torque (F/T)

With Muse, touch learned later improves skills learned earlier — a small amount of F/T data on new tasks improves zero-shot on diverse pretraining tasks that were never supervised with F/T

We believe MuSe provides a practical pathway towards training multisensory foundation models that leverage both abundant vision data, and smaller multisensory datasets 🧵👇

[Jaden Clark @jadenvclark]

Naive finetuning forgets old tasks. We finetune with 2⃣ ER to preserve visual and task generalization and 3⃣ multistage fusion to amplify the new modality

[Jaden Clark @jadenvclark]

How does MuSe work? We outline 3 key components:
1⃣ Multisensory future prediction (world modeling): Training the model to predict future actions AND multisensory observations encourages it to learn shared representations

[Stanford MSL @StanfordMSL]

π, But Make It Fly ✈️
We fine-tuned π0, a VLA model pretrained entirely on manipulators, to fly a drone that picks up objects, navigates through gates, and composes both skills from language commands.

[Xiaomeng Xu @XiaomengXu11]

Can we learn whole-body mobile manipulation directly from human demonstrations?

Introducing Whole-Body Mobile Manipulation Interface (HoMMI)

Egocentric + UMI, 0 teleop -> bimanual & whole-body manipulation, long-horizon navigation, active perception

https://t.co/CcZ9ZwfuFr

[Zhanyi Sun @s_zhanyi]

We find that RL post-training can substantially improve BC policies without teaching them anything fundamentally new.

So what is RL doing? In DICE-RL, it contracts a broad behavior prior toward high-value modes. (1/n)

https://t.co/5WbSgSQ5Ok

[Zeyi Liu @Liu_Zeyi_]

For video generation in robotic applications, looking pretty is usually not enough.
Robot manipulation requires understanding how visual observations and 3D geometry evolve over time under agent actions, with temporal coherence and geometric consistency across camera views.

We study this challenge in our work (recently accepted by @iclr_conf ), 4D Video Generation for Robot Manipulation, which enforces multi-view 3D consistency via geometric supervision to generate spatio-temporally aligned videos.

[Moo Jin Kim @moo_jin_kim]

We release Cosmos Policy 💫: a state-of-the-art robot policy built on a video diffusion model backbone.

- policy + world model + value function — in 1 model
- no architectural changes to the base video model
- SOTA in LIBERO (98.5%), RoboCasa (67.1%), & ALOHA tasks (93.6%)

🧵👇

[Jaden Clark @jadenvclark]

A key aspect of scaling robot data collection is sensor reliability (hence why we haven't really seen tactile sensing at scale yet). UMI-FT addresses this by giving UMI robust/reliable fingertip level force-torque sensing, a major step toward scaling data for contact-rich tasks.

[Hojung Choi @Hojung_Choi_]

Robots excel at learning motions from humans, but can they also learn to apply force safely? 💪

Introducing UMI-FT: the UMI gripper equipped with force/torque sensors (CoinFT) on each finger. Multimodal data from UMI-FT, combined with diffusion policy and compliance control, enables robots to apply sufficient yet safe force for task completion.

UMI-FT Project website: https://t.co/8C3bI79uhN
CoinFT Project website: https://t.co/vSLBzg5HaC

[Toby Mao @tobymm25]

In Silicon Valley, “virtual cells” are suddenly everywhere.

Meta and CZI recently went all in, signaling that this is no longer a fringe research direction.

So what is virtual cell? Check out my new blog about virtual cell: https://t.co/lQx2dc4XsZ

[Jaden Clark @jadenvclark]

Simple heuristics can make a big difference when using foundation models for science. See our new paper on tracking animals with SAM 2, and check out some of the conservation work we're doing with it down in Costa Rica: https://t.co/xWrVfF5p9q

[Methods in Ecology and Evolution @MethodsEcolEvol]

📖Published📖

Lalgudi et al. introduce Frame-Level Alignment and Tracking (FLAIR). FLAIR takes a drone video as input and outputs segmentation masks of the species of interest across the video 🦈

https://t.co/WNH3gRmI0t