Jimmy (Tsung-Yen) Yang

Researcher at @Waymo Research; ML/Embodied AI; PhD @Princeton ECE

[Basile Terver @BasileTerv987]

📢 Accepted to TMLR, with reproducibility certification 🏅

v2 of our JEPA-WM study (arXiv:2512.24497) is out, with new data-scaling experiments, a Lipschitz analysis of multistep rollout training, and extended discussions.

Recap + what's new 👇

w/ @JimmyTYYang1, Jean Ponce, @AdrienBardes, @ylecun

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

https://t.co/G2dbfWwxuz

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

A career update, and let the journey begin!

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

It was great to work with @BasileTerv987 on this project! The JEPA world model works well on unseen robot tasks. Thanks Basile for the amazing work!

[Basile Terver @BasileTerv987]

My first PhD paper is out! 🎓

"What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?"

tl:dr: JEPA-WMs for robotics: learn dynamics on top of visual encoders, optimize actions towards goal 👇

w/ @JimmyTYYang1, Jean Ponce, @AdrienBardes, @ylecun

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

It is my pleasure to work with Raktim! A concrete example of how to use human videos to learn robot policies

[Raktim Gautam Goswami @raktimgg]

Introducing DexWM: Dexterous Manipulation World Model

tl;dr: train on human videos; fine-grained actions; hand-consistency loss; DexWM+MPC -> zero-shot dexterous manipulation

w/ @_amirbar, @DavidJFan, @JimmyTYYang1, @GaoyueZhou, P. Krishnamurthy, M. Rabbat, F. Khorrami, @ylecun

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

@chris_j_paxton love this!

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

@chris_j_paxton this is so cool! were you able to see the robot doing the demo that they showed in the video?

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

@su_lin_liu hi sir!

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

It was so much fun to work on this: llama3 + sim2real skills, all are open-sourced!

[Dhruv Batra @DhruvBatra_]

FAIR researchers (@AIatMeta) presented SegmentAnything and our robotics work at the White House correspondents’ weekend.

Llama3 + Sim2Real skills (trained with @ai_habitat) = a robot assistant

[Dhruv Batra @DhruvBatra_]

FAIR researchers (@AIatMeta) presented SegmentAnything and our robotics work at the White House correspondents’ weekend.

Llama3 + Sim2Real skills (trained with @ai_habitat) = a robot assistant

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

@chris_j_paxton indeed!

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

@chichengcc very cool! How long does it take to collect these demo data?

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

@PeterMitrano @MarkVanderMerwe very cool demo! We are also building some infra for Spot robot: https://t.co/jgh78pNjkH

Using this could speed up image collection!

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

Very excited to see that Habitat 3.0 is finally out! Please also check out our paper here: https://t.co/Hrt305Cz8g

[Dhruv Batra @DhruvBatra_]

Announcing Habitat 3.0, simulating humanoid avatars and robots collaborating!

- Humanoid sim: diverse skinned avatars
- Human-in-the-loop control: mouse/keyboard or VR
- Tasks: social navigation and rearrangement

Over 1,000 steps per second on 1 GPU for large-scale learning!

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

ASC: https://t.co/9GW1Vnyd9X
LSC: https://t.co/orL7B0t7L6

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

Please check out our Spot Sim2Real repo! https://t.co/jgh78pNjkH

It aims to reproduce the demos of ASC and LSC. We open-source the weights of the navigation, picking, and place skills, and the infrastructure to use Large Language Models (LLMs) to interact with those skills!

[AI Habitat @ai_habitat]

Habitat stable version v0.2.5 released!

-- 🤖 Hardware bridge for BD Spot
-- 📷 Improved PBR rendering for better synthetic scene visuals
-- 🤗 ai-habitat on HuggingFace
-- 🏠 Habitat Synthetic Scenes Dataset (HSSD)

and more...
https://t.co/QOteNX9SSK
https://t.co/kLrsMC057R

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

@chris_j_paxton I saw this too!!! It will be big and transformative if this is true! Look forward to seeing other researchers to verify this

[Boston Dynamics @BostonDynamics]

“We were very interested in testing out high-level reasoning and planning. We wanted to see...if the robot could figure out the steps it needed to take to solve the problem.” - Discover why @MetaAI chose Spot for their research. https://t.co/liLiiARsYd

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

This is our recent work building toward "reproducible" robotic research. Thanks @chris_j_paxton for leading such an amazing project!

[Chris Paxton @chris_j_paxton]

The future of robot butlers starts with mobile manipulation.
We’re announcing the NeurIPS 2023 Open-Vocabulary Mobile Manipulation Challenge!
- Full robot stack ✅
- Parallel sim and real evaluation ✅
- No robot required ✅👀

https://t.co/mggAbRhrLP

[Jimmy (Tsung-Yen) Yang @JimmyTYYang1]

our demo @CVPR!

[Dhruv Batra @DhruvBatra_]

FAIR and ⁦@gtcomputing⁩ researchers demoing our sim2real work on ⁦@BostonDynamics⁩ Spot at ⁦@MetaAI⁩ booth at #CVPR23.