Jiahua Dong

Ph.D. student @ UIUC

[Jiahua Dong @JiahuaDong]

@li_lkhlkh329 It's TI2V with additional action conditioning. The model takes an initial RGB frame, a text description, and an agent-motion trajectory to predict future frames. See Sec. 3.1 for details.

[Jiahua Dong @JiahuaDong]

[1/7] We trained a world model on 30,000 hours of first-person video.

It learned exactly where the hands go. What happens to the things they touch, much less so.

Going from 300 to 30,000 hours did not narrow the gap, and a 4× larger model barely did either.

[Jiahua Dong @JiahuaDong]

@JulienBlanchon We train for 1 epoch over the 30k-hour dataset (~65k optimization steps, global batch size 512). Please see Appendix A.4 for training details.

[Jiahua Dong @JiahuaDong]

@s_arkhangelskiy Thanks so much for your interest! We haven’t trained or adapted our model on DROID, so we don’t currently have a DROID checkpoint compatible with PhAIL’s setup.

[Jiahua Dong @JiahuaDong]

@TongheZhang01 Partial observability is an important part of the picture. More video alone might not suffice, but combining learned models with simulators could help us make better use of the data through physical priors, while accounting for uncertainty about hidden states.

[Jiahua Dong @JiahuaDong]

@tarat_211 Thanks for your interests!

We use 64 H100 for Cosmos 3 Super and 32 H100 for Cosmos 3 Nano. Overall it takes about 8000 GPU hours for the Super model and 3000 GPU hours for the Nano model.

[Jiahua Dong @JiahuaDong]

Thanks to the team and collaborators!

@ptokmakov @Miccooper9 @mzubairirshad @off_jangir @ZakharovSergeyN @mangahomanga @vitorguizilini

@ToyotaResearch @CarnegieMellon @JohnsHopkins @UofIllinois

We'd love to hear your thoughts on the work!

[Jiahua Dong @JiahuaDong]

[7/7] Takeaways

Scaling ego-centric video helps and transfers to robots. But contacts, forces and hidden geometry are never visible in pixels — precisely what simulators represent well. Combining the two may be what’s needed.

Code, checkpoints and our OOD benchmark coming soon.

Project: https://t.co/6Vdz1M8SPm
Paper: https://t.co/GddLekoUyV