Zhiyi Li

PhD student at MIT EECS, 3D vision, robotics, and generative world models.

[Chloe Su ✈️ NeurIPS 2026 @Huangyu58589918]

What if multimodal generation didn’t have to choose between understanding and creating, or between images and text going first?

Excited to share JUICE 🧃 with @JiahanCed and the team!

What I find especially interesting is the role of representations: the same features that the model uses to understand an input also become the states it learns to generate. Images and text can then take shape together, with each modality providing context for the other throughout generation.

One detail from our ablations: even when only one modality is supervised on joint examples, keeping the noisy partner stream improves image generation over conditional-only training at the same target count. Cross-modal interaction can be useful before either side is fully formed.

The maze experiments make this tangible. A drawn route and a written sequence of moves can each be valid while describing different solutions. Coupled generation lets the two streams settle on a consistent pair, without choosing in advance which modality goes first.

This also raises an interesting question for multimodal memory: could similar interactions help models recover useful context from incomplete visual and linguistic cues?

Excited to explore where this direction leads!

[Runhan Huang @RunhanH]

🚀 RAX now supports DeepSeek Harness and Kimi Code as agent harnesses!

We’ve also added direct support for self-hosted DeepSeek-V4.1-Flash, so you can run RAX with local inference.

On the DexJoCo benchmark, RAX + DeepSeek-V4.1-Flash outperforms π0.5 and Fast-WAM by more than 10% with just 3 demonstrations. At API pricing, it costs roughly 1/20 as much as Claude Opus 5.5!

Try it out! 🤖

🌐 Project + sessions:
https://t.co/yS0mTf7T96
📄 Paper:
https://t.co/BSRB3PQjLb
💻 GitHub:
https://t.co/XGLeyZaz5v
🧩 Skill library:
https://t.co/tcbFIherKu
🤗 Dataset with full robot logs and sessions:
https://t.co/wXsHJ3TVfp

@deepseek_ai @Kimi_Moonshot

#Robotics #EmbodiedAI #AIAgents #OpenSource #DeepSeek #RobotLearning #LLM

[Yilun Du @du_yilun]

Robotics has spent years scaling demonstration data and model size. What happens when we scale EXPLORATION for robot agents?

We're incredibly excited to share RAX, an agentic robotics system that learns by scaling online exploration.

A coding agent connects directly to the robot’s perception and control APIs, learns from feedback, and builds programs that perceive, plan, and act. The model stays frozen. As exploration scales, the agent improves its programs and builds up reusable skills.

The results:
• ZERO teleoperation or action demonstrations on real robots—from insertion with 0.1 mm clearance and air-fryer manipulation to deformable objects and long-horizon mobile whole-body control.
• +37 percentage points on average over leading learned policies across five simulation benchmarks: RoboDojo, BEHAVIOR-1K, HumanoidArena, DexJoCo’s bimanual tasks, and LIBERO-Pro.
• One framework spanning single-arm, bimanual, dexterous, humanoid, and mobile robots, with new embodiments connected through APIs.

The craziest part: on each real-world embodiment, going from scratch to mastering the tasks took roughly two hours. No teleop. No model finetuning.

We’re open-sourcing the full framework AND every experiment session. We can’t wait to see what people build with it!

Project + sessions: https://t.co/O04Z5PUgqY
Paper: https://t.co/sd66qnYZWB
GitHub: N

Project fully led by @RunhanH and @xczhangxx who are absolutely awesome.

How it works, experiments, and robot videos below 🧵

(1/N)

[Zhiyi Li @ZhiyiLi02]

Excited to share our work on MoSE3, accepted to #NeurIPS2026 Spotlight!

Great collaboration with @joannac03559818 and @TianX_ia , and many thanks to @ruojin8 , @du_yilun and @QianqianWang5 for the guidance!

[Joanna Jiahuan Cheng @joannac03559818]

Excited to share that MoSE3 was accepted as #NeurIPS2026 Spotlight!

MoSE3 estimates an SE(3) transform at every pixel from monocular video, all in a shared world coordinate frame and in a single forward pass. It captures rigid, articulated, and deformable motion.

1/

[Chloe Su ✈️ NeurIPS 2026 @Huangyu58589918]

Excited to share our new work: TabFM-Auto at @GoogleResearch & @GoogleDeepMind ! 🚀
Freeze the model. Evolve the pipeline.
An LLM agent improves the data pipeline around a frozen tabular foundation model—gaining 228 Elo and ranking #1 overall across 51 TabArena datasets.

Paper: https://t.co/t89gSxbwcB
Project + interactive results: https://t.co/DIZ05cvDoa

[Chloe Su ✈️ NeurIPS 2026 @Huangyu58589918]

📢 Deadline Extension: To avoid clashing with the ICLR deadline, we are extending our submission deadline to September 30 (AoE)!
Take an extra week to polish your work. Looking forward to your submissions! 📄✨ @corl_conf

[Chloe Su ✈️ NeurIPS 2026 @Huangyu58589918]

📣 CALL FOR PAPERS 📣

𝗖𝗼𝗥𝗟 𝟮𝟬𝟮𝟲 𝗪𝗼𝗿𝗸𝘀𝗵𝗼𝗽 𝗼𝗻 𝗠𝗲𝗺𝗼𝗿𝘆 𝗳𝗼𝗿 𝗥𝗼𝗯𝗼𝘁 𝗙𝗼𝘂𝗻𝗱𝗮𝘁𝗶𝗼𝗻 𝗠𝗼𝗱𝗲𝗹𝘀 @corl_conf

📅 Submission deadline: September 24, AoE
📢 Notifications: October 12
📍 Workshop: November 12 in Austin
📝 Submit: https://t.co/DVE3ZfPQ6C

Robots operating over long horizons need to remember: which drawers they have already opened, why the last grasp failed, or how a particular user likes their kitchen arranged. But what should memory mean for a robot? What should it store, and how can we tell whether it truly helps—or whether we have simply given the model a larger context window?

If you are working on or thinking about these questions, we would love to see your work. Our goal is to map the design space for memory in robot foundation models, and 𝘄𝗲 𝗮𝗰𝘁𝗶𝘃𝗲𝗹𝘆 𝗶𝗻𝘃𝗶𝘁𝗲 𝗽𝗼𝘀𝗶𝘁𝗶𝗼𝗻 𝗽𝗮𝗽𝗲𝗿𝘀, 𝘄𝗼𝗿𝗸 𝗶𝗻 𝗽𝗿𝗼𝗴𝗿𝗲𝘀𝘀, 𝗮𝗻𝗱 𝗽𝗿𝗲𝗹𝗶𝗺𝗶𝗻𝗮𝗿𝘆 𝘀𝘁𝘂𝗱𝗶𝗲𝘀. 𝗔𝗰𝗰𝗲𝗽𝘁𝗲𝗱 𝗽𝗮𝗽𝗲𝗿𝘀 𝗴𝗲𝘁 𝗮 𝗽𝗼𝘀𝘁𝗲𝗿, 𝘄𝗶𝘁𝗵 𝗮 𝗳𝗲𝘄 𝘀𝗲𝗹𝗲𝗰𝘁𝗲𝗱 𝗳𝗼𝗿 𝗼𝗿𝗮𝗹𝘀.

Learn more at https://t.co/NWWWaRJqe4

Organized with @pgozdil @Weirui_Ye @tarikkelestemur @JNZaech @fpaissan_ @ZhiyiLi02 @du_yilun

#CoRL2026 #RobotLearning #Robotics #EmbodiedAI #FoundationModels

[Zhiyi Li @ZhiyiLi02]

Many thanks to my advisor @du_yilun and all coauthors @PeilinWu64 @xshenhan @ruojin8!

[Zhiyi Li @ZhiyiLi02]

🚀 New work [ICML 2026]: Structured 4D Latent Predictive Model for Robot Planning.

Can robots plan by imagining future 3D structure, not just pixels?

Multi-view obs + language → 3D latent rollouts → 3D-consistent subgoals for robot actions.
https://t.co/9qm9mzMoFO

[Zhiyi Li @ZhiyiLi02]

(7/7) Takeaway: future 3D prediction is a useful bridge between generative world models and robot control.
Please visit our project page for more results, visualizations, and details: https://t.co/dg0pgksfkx.

[Zhiyi Li @ZhiyiLi02]

(6/7) Real world: we train from block-in-basket demos with 4 RGB-D cameras.
Predicted 3D subgoals are registered to recover gripper poses, then executed with training-free point cloud registration-based motion planning, showing the pipeline can transfer beyond simulation.

[Zhiyi Li @ZhiyiLi02]

(5/7) Generalization: for robot planning, small unseen changes—lighting, noise, background color, or camera pose—can break policies.
Structured 3D rollouts make the planner more robust under these shifts.

[Zhiyi Li @ZhiyiLi02]

(4/7) Generation results: the model unrolls language-conditioned 4D futures across tasks, from stacking and insertion to longer-horizon manipulation.
Rendered views and point clouds stay coherent over time, showing strong 3D consistency.

[Zhiyi Li @ZhiyiLi02]

(3/7) How it works: reconstruct current 3D latent, use a Single Dynamics Model for coarse structure and a Latent Generator for details, then decode future latents into point clouds/3D Gaussians, which are then converted to robot actions by inverse dynamics.

[Zhiyi Li @ZhiyiLi02]

(2/7) Key idea: predict future 3D structure directly.
Multi-view observations are encoded into a structured 3D latent, then rolled forward over time conditioned on text instructions.

[Zhiyi Li @ZhiyiLi02]

(1/7) Why 3D in planning? A general-purpose robot in the real world must reason about where objects are, how they move, and how actions change the scene.
2D pixels can look plausible but be geometrically inconsistent. For control, predictions need 3D structures.