📢 Accepted to TMLR, with reproducibility certification 🏅
v2 of our JEPA-WM study (arXiv:2512.24497) is out, with new data-scaling experiments, a Lipschitz analysis of multistep rollout training, and extended discussions.
Recap + what's new 👇
w/ @JimmyTYYang1, Jean Ponce, @AdrienBardes, @ylecun
My first PhD paper is out! 🎓
"What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?"
tl:dr: JEPA-WMs for robotics: learn dynamics on top of visual encoders, optimize actions towards goal 👇
w/ @JimmyTYYang1, Jean Ponce, @AdrienBardes, @ylecun
Introducing DexWM: Dexterous Manipulation World Model
tl;dr: train on human videos; fine-grained actions; hand-consistency loss; DexWM+MPC -> zero-shot dexterous manipulation
w/ @_amirbar, @DavidJFan, @JimmyTYYang1, @GaoyueZhou, P. Krishnamurthy, M. Rabbat, F. Khorrami, @ylecun
FAIR researchers (@AIatMeta) presented SegmentAnything and our robotics work at the White House correspondents’ weekend.
Llama3 + Sim2Real skills (trained with @ai_habitat) = a robot assistant
FAIR researchers (@AIatMeta) presented SegmentAnything and our robotics work at the White House correspondents’ weekend.
Llama3 + Sim2Real skills (trained with @ai_habitat) = a robot assistant
@PeterMitrano@MarkVanderMerwe very cool demo! We are also building some infra for Spot robot: https://t.co/jgh78pNjkH
Using this could speed up image collection!
Announcing Habitat 3.0, simulating humanoid avatars and robots collaborating!
- Humanoid sim: diverse skinned avatars
- Human-in-the-loop control: mouse/keyboard or VR
- Tasks: social navigation and rearrangement
Over 1,000 steps per second on 1 GPU for large-scale learning!
Please check out our Spot Sim2Real repo! https://t.co/jgh78pNjkH
It aims to reproduce the demos of ASC and LSC. We open-source the weights of the navigation, picking, and place skills, and the infrastructure to use Large Language Models (LLMs) to interact with those skills!
“We were very interested in testing out high-level reasoning and planning. We wanted to see...if the robot could figure out the steps it needed to take to solve the problem.” - Discover why @MetaAI chose Spot for their research. https://t.co/liLiiARsYd
The future of robot butlers starts with mobile manipulation.
We’re announcing the NeurIPS 2023 Open-Vocabulary Mobile Manipulation Challenge!
- Full robot stack ✅
- Parallel sim and real evaluation ✅
- No robot required ✅👀
https://t.co/mggAbRhrLP