Omar Rayyan

PhD Student at @GoogleDeepMind @UCLA

[Omar Rayyan @orrayyan]

Meet FetchMan: a vision-based humanoid policy trained entirely in simulation that transfers zero-shot to diverse real-world scenes and objects.

Simulation has produced impressive locomotion policies that transfer to the real world. We wanted to see how far the same recipe goes for vision-based loco-manipulation. More below🧵

[Omar Rayyan @orrayyan]

Training for the robot olympics with MuJoCo’s new surfacevel for geoms

[Omar Rayyan @orrayyan]

@kevin_zakka Very useful!!

[Omar Rayyan @orrayyan]

@zhengyiluo Thanks! powered by SONIC

[Zhengyi “Zen” Luo @zhengyiluo]

A solid step toward humanoid loco-manip, super cool!

[Mahi Shafiullah 🏠🤖 @notmahi]

Learning mobile manipulation is hard because collecting data for mobile manipulation is hard. But seems like @omarrayyann has found a promising way to crack this problem!

[Omar Rayyan @orrayyan]

@XiaoChen_Twi thanks!

[Omar Rayyan @orrayyan]

@JulienBlanchon yes will be uploading it to hf

[Omar Rayyan @orrayyan]

Meet FetchMan: a vision-based humanoid policy trained entirely in simulation that transfers zero-shot to diverse real-world scenes and objects.

Simulation has produced impressive locomotion policies that transfer to the real world. We wanted to see how far the same recipe goes for vision-based loco-manipulation. More below🧵

[Omar Rayyan @orrayyan]

@JulienBlanchon Thanks! code will be out soon. but yes, per iteration I sample 64 tasks and roll each 8 times from the same starting state. noise is injected at a single random flow step and logp is taken there. and thanks for the suggestion, I will be trying it out.

[Omar Rayyan @orrayyan]

Check out our website for more videos and interactive simulation episodes: https://t.co/XoP6vojZNZ.

This project was done with advice from @YuchenCui1 and great collaborators @max_argus, Zhi Li, Chang Yu, Yuxin @chenfanfujiang.

Paper: https://t.co/6QaTfVE2f2
Code (soon): https://t.co/if6SiW6ypL

[Omar Rayyan @orrayyan]

We follow this sim-to-real recipe to train two policies: a single-object Fetch policy that works across scenes, and an object-conditioned policy that takes the target object's name as an instruction. Both are two-system architectures with the decoupled SONIC controller acting as the low-level controller.

[Mahi Shafiullah 🏠🤖 @notmahi]

We will be in the #RSS2026 poster session at 6:30 PM – stop by with all of your questions!

[Omar Rayyan @orrayyan]

I’m at RSS in Sydney 🇦🇺 to present:

- MolmoSpaces, Tuesday 6:30pm
- Contact Anchored Policies, Wednesday 4:00pm
- a (tbr) work on training general visual loco-manipulation policies in sim.

If you work on anything from learning with off-domain data to sim-evals, let’s chat!

[Omar Rayyan @orrayyan]

@kevin_zakka @pthangeda_ are we thinking of the same thing

[Omar Rayyan @orrayyan]

@FirasBDarwish Cool work firas!

[Mahi Shafiullah 🏠🤖 @notmahi]

Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data?
Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧵