Omar Rayyan

phd student @GoogleDeepMind @UCLA

[Omar Rayyan @omarrayyann]

Meet FetchMan: a vision-based humanoid policy trained entirely in simulation that transfers zero-shot to diverse real-world scenes and objects.

Simulation has produced impressive locomotion policies that transfer to the real world. We wanted to see how far the same recipe goes for vision-based loco-manipulation. More below🧵

[Omar Rayyan @omarrayyann]

@JulienBlanchon Can you elaborate?

[Omar Rayyan @omarrayyann]

@Nicolas_Keller You should make it more clear that these are not the results of the base released policies, but a fine-tuned version of them

[Omar Rayyan @omarrayyann]

@georgiagkioxari @notmahi we released a dataset previously that is within-domain to our benchmark (https://t.co/cBZzeFsJTt). but others got such results with Astra in real-world evals so I wouldn't attribute the performance to this dataset existing

[Omar Rayyan @omarrayyann]

@SOTA_kke they can also just add them to the pool of rl envs they use for post-training. maybe they already do that

[Omar Rayyan @omarrayyann]

@NicholasEPfaff @RussTedrake @phillip_isola Congrats!

[Mahi Shafiullah 🏠🤖 @notmahi]

Are we experiencing a GPT moment in robotics? Our MolmoSpaces benchmark seems to suggest so. Check out @omarrayyann's thread to see the details 👇

[Omar Rayyan @omarrayyann]

I think distilling the model’s spatial knowledge into a raw action space policy would be an interesting direction. its good vision understanding makes this verifiable enough to happen in an automated closed-loop way

[Omar Rayyan @omarrayyann]

GPT-Astra outperformed all open-source VLA/WAM baselines on a subset of our zero-shot MolmoSpaces-v1 benchmark! traces: https://t.co/8Le4Pni17H

[Omar Rayyan @omarrayyann]

It used fewer reasoning tokens than other VLM APIs (though the counts might not be directly comparable). I couldn’t run the full benchmark for these API models to include them on our leaderboard (https://t.co/d7RlL6gAa6) because of the API cost :)

[Omar Rayyan @omarrayyann]

@kaiwynd how much did this cost

[Omar Rayyan @omarrayyann]

Are you learning the noise of size action_dim*chunk_size? If so, 40 episodes might be too little for that. For real-world experiments, they learn a noise of size action_dim and tile it across the chunk dimension. Should be more sample efficient (assuming you have no bugs) given the simple pick task you have.

[Omar Rayyan @omarrayyann]

Training for the robot olympics with MuJoCo’s new surfacevel for geoms

[Omar Rayyan @omarrayyann]

@kevin_zakka Very useful!!

[Omar Rayyan @omarrayyann]

@zhengyiluo Thanks! powered by SONIC

[Zhengyi “Zen” Luo @zhengyiluo]

A solid step toward humanoid loco-manip, super cool!

[Mahi Shafiullah 🏠🤖 @notmahi]

Learning mobile manipulation is hard because collecting data for mobile manipulation is hard. But seems like @omarrayyann has found a promising way to crack this problem!

[Omar Rayyan @omarrayyann]

@XiaoChen_Twi thanks!

[Omar Rayyan @omarrayyann]

@JulienBlanchon yes will be uploading it to hf

[Omar Rayyan @omarrayyann]

Meet FetchMan: a vision-based humanoid policy trained entirely in simulation that transfers zero-shot to diverse real-world scenes and objects.

Simulation has produced impressive locomotion policies that transfer to the real world. We wanted to see how far the same recipe goes for vision-based loco-manipulation. More below🧵