Yu Lei

PhD student @UTCompSci | Learn to understand ourselves and build intelligence.🤖🧠👁️

[Yu Lei @_OutofMemory_]

🤖Co-training is everywhere (sim↔real[e.g. GR00T, LBM], human↔robot[e.g. PI, EgoScale], even non-robot data[e.g. PI, LBM).
But why does it work? How can we improve it further?

Taking sim-and-real imitation learning in diffusion/ flow-based models as the test bed, we performed a rigorous mechanistic analysis, drawing on theoretical insights and multi-layered experiments.

😮Key insight: it’s all about representations.
- Alignment → enables transfer
- Discernibility → enables adaptation

⚖️Both are necessary — it's better to have more aligned representations, but the model must be able to discern the domains. We term this as structured representation alignment.

⬇️Let’s take a deep dive into that:
Paper: https://t.co/RWCAxdBC0j
Website: https://t.co/BwgbwCkevA

[Yuke Zhu @yukez]

Something exciting is coming next Monday 👀

[Jim Fan @DrJimFan]

CoRL Keynote. Hilton Austin. Monday Nov 9, 1:45 pm. 2 days countdown.

Something extraordinary.

[Yu Lei @_OutofMemory_]

Excited to welcome y'all coming to Austin to attend the amazing #CoRL2026 🤠.

Looking forward to catching up with old friends and meeting new ones!

Been thinking about representation learning and (mental) simulation in the era of agentic robotics, multimodal pre/co-training, and deep learning science.

Please reach out if you'd like to chat!!

[Yu Lei @_OutofMemory_]

Insightful study as always.

[Younghyo Park ✈️CoRL @younghyo_park]

Robot arms can be extremely heavy compared to the objects they are asked to manipulate, especially with their metal links and highly geared motors.

But is that actually a problem for robot learning, or are learning algorithms powerful enough to handle it? Are we paying any hidden costs for using heavy robots? Here's what we found 🧵

[Liqian Ma @LiqianMa4]

In sim-and-real co-training, robots learn from a few real demos plus ample sim data. But which sim data should we generate?

MetaGenSim meta-learns what to simulate: 2.5× the real-world success of uniform sim data, matching expert hand-tuning.

https://t.co/ZLy0PeJ8Ln
#CoRL2026

[Shuran Song @SongShuran]

The science of pretraining deserves more attention!

What makes a good pretrained model? How to evaluate the it? When should we stop training? And how do we optimize pretraining for downstream SFT or RL?

Some answers may be counterintuitive, e.g., the best-performing checkpoint may not be the best starting point for downstream learning.

If you're curious now, check out @s_zhanyi's work “Beyond Validation Loss” 👇

[Tony Tao @ CoRL @_tonytao_]

Back in ~Dec 2025, I really wanted to work on mobile manipulation, but robots were either too expensive or not performant enough.

So we built our own!

Excited to finally share something I’ve had so much fun working on:

1. Pixel: a co-designed full stack system for learning mobile manipulation.
2. Pixel-UMI: environment aware whole body manipulation without robot data.

w/ @JasonJZLiu

(1/N)

[Zhanyi Sun @ CoRL @s_zhanyi]

Can more pretraining improve a robot policy’s success rate, yet makes it worse for porttraining? 🤔

In our new work, we find that it can. Continued supervised pretraining of generative robot policies can improve reliability at the expense of coverage, and this tradeoff affects downstream posttraining.

Read on to see why this happens and how we can preserve coverage. 1/n 🧵

[Alex Krentsel @AlexKrentsel]

i've reproduced the results from the MIT blog post today on weight-space programming. but one additional result i have to add: this is a new, step-change emerging capability that's appeared in just the latest set of models.

[Yu Lei @_OutofMemory_]

@WinstonGu_ looks cool!

[Jimmy Wang @jwang771]

We introduce 3D World Simulator, an interactive world model that runs at 50+ FPS on an RTX 5090.

Our model predicts in a unified 3D representation space, capable of generating:
• Novel view appearance
• Multi-view consistent RGB-D
• 3D geometry
• Spatially aligned tactile signals

Play with the world model now at https://t.co/eveClAxURp!

[Yu Lei @_OutofMemory_]

You definitely want a world model that do efficient test-time-adaptation. Please check out @yluo_y’s nice work!

[Yunhao Luo @yluo_y]

🌏 World models need to capture a scene’s hidden physics properties, e.g., mass or friction, to generate predictions consistent with physical reality.

🚀 Excited to introduce PhysiCalWorld: In-Context Physics Calibration for World Models! We calibrate world models through an interaction clip, grounding future predictions in scene-specific dynamics.

🏷️ Supervision: trained without physics labels
⚡ Efficiency: minimal additional inference cost
⚛️ Generalization: extrapolation beyond the physics ranges seen during training

https://t.co/k9PRgg7qmR

More below 🧵

[Yu Lei @_OutofMemory_]

@phillip_isola @randall_balestr @ylecun 😺

[Randall Balestriero @randall_balestr]

The Platonic Representation Hypothesis is now a Theorem ◼️
Train JEPAs separately, on any modalities: they provably learn the same representation up to one rotation.
The result? A 109M JEPA-Text model that aligns with vision models better than 72B LLMs.
w/ @ylecun 🧵

[Yiming Dou @_YimingDou]

Introducing DreamFAR, a world model trained from scratch using only robot data, without internet pretraining. It supports:
🎮 Real-time teleoperation
⏱️ Minutes of long-horizon rollout
🧩 Generalization across 1,000+ tasks

https://t.co/A1ifHlj69O

🧵 (1/8)

[Yu Lei @_OutofMemory_]

@MatthewMao Congrats Mao!

[Google DeepMind @GoogleDeepMind]

This is Genie 4, our most realistic and interactive world model yet. 🌐

Explore dynamic environments in real time to see how the scene responds, sounds, and how the people in it move, react, and speak.

[Yu Lei @_OutofMemory_]

@ChongZzZhang @PranaLabs_ oh wow

[imjustnewatai @imjustnewatai]

Evil qwen

[Yinghao Xu @YinghaoXu1]

🎉 After one year of robot pretraining, we learned one thing the hard way: data is everything.
Today we're excited to release FOMI, a Force-feedback Manipulation Interface for teaching contact-rich manipulation!

Inspired by @SongShuran's UMI, FOMI keeps handheld, robot-free data collection and adds what operators have been missing: you can feel what the gripper touches ✋ No force sensors, no motors.

Two key results:
🔑 29% more successful demos than UMI in the same time, and 18% more than GELLO in teleoperation
🔑 π0.5 fine-tuned on about an hour of FOMI data per task reaches 78.4% average success on four contact-rich tasks

Let's check out our demos 👇