Imagine you have three robot policy checkpoints saved from training, and one robot.
Many labs train checkpoints faster than they test them. Slow evaluation delays reliable robots.
I think the first broadly useful job for robot world models will be evaluation, not control.
@randall_balestr@KerdaVaraku@ylecun The failure at discontinuous contacts you characterize matters a lot for manipulation. X Square saw the same in world-model policy eval, their model overestimated progress most through contact, with the peak at insertion.
@theSamPadilla A benefit of gaming data is that the engine logs the exact actions. We trained an inverse dynamics model only on games to label camera motion in real video, and it got 86% of real clips right reading optical flow. So it transfers if you strip the game look.
@deepakpathak How do you track the gap closing week to week? Waymo doesn’t measure the gap directly, but they require the sim to be no easier than reality. On ~70 scenarios replayed on a test track, the sim has to show as many or more collisions as the track did.
@off_jangir Did you try ranking a few policies in it against their real robot ranking? That matters for eval imo. X Square did it with 5 checkpoints and kept 89% of the real pairwise orderings.
@jerthesquare_ Saw the same thing for us with an inverse dynamics model. We trained it on games to label camera motion in real video, and on real clips raw pixels got 31% right vs 86% from optical flow. So across a domain shift the gap is much bigger than in your in-distribution probe.
@firatcand@100illi@chris_j_paxton In crowd labeling the check for this is overlap. Several annotators get the same item, if one differs it’s the annotator, if they all split the instruction is unclear. Might work for teleop too, several operators on the same task from the same start.
@du_yilun I like that data without depth labels still improves depth here. Dyna-2 saw a similar behavior with actions, adding video with no action labels cut the action error on unseen robot data from 0.34 to 0.12.
@Builtin_Pb I think Habitat had a similar problem in 2020, policies learned to slide along walls which the real robot can’t do obviously. Sim vs real correlation was 0.18, and after fixing the sim it went to 0.844 with the same policies.
@aimalysheva@ArtificialAnlys well yes the providers pick specific model configuration to serve, from quantization and batch size to other parameters... Gotta check before using.
@gurtej__gill_ I saw similar behavior with LLM alignment and directionally similar with video models, making a diverse and high quality SFT data isn't as hard, but it yields really good results without unnecessary complexity.
@KnightNemo_@zhaohang0124 I also showed that the inference speed of world-action models can be as fast as that of VLAs, with some optimizations of the action inference: 53 milliseconds for Faster-WAM versus 65 milliseconds for the π0 VLA. Full post: https://t.co/2uBT6OFrEe
In my previous post I mapped five different designs for feeding actions into world models. In my new post I go deeper into the third design, where actions get their own layers, and show how they meet the video in the attention step.
@KnightNemo_ Whether to use the videos for prediction or not depends on the setting. Skipping them works much faster, 190ms against 580ms in Fast-WAM (@zhaohang0124 and others), and is fine on familiar scenes, but so far it gives worse results for novel scenes.
@nathanbenaich@RaiaHadsell@GoogleDeepMind@raais This works from humans to robots too. In NVIDIA’s EgoScale the robot learned to fold a shirt from one robot demo and a few human ones, with 88% success. I think only about 4 hours of robot data was needed.
@agilityrobotics Imo this is the case where generating the video at inference is worth it. On familiar scenes it barely matters, but MoWAM got +10 points on LIBERO-Plus with changed cameras and lighting.
@chris_j_paxton Self-driving had the same problem in the past. Waymo builds scenarios from crash records and replays them on each candidate release. Feels like robots need the same loop.