China open-sourced a model that reconstructs any scene in 3D from a regular video, in real-time.
one camera. no LiDAR. 10,000+ frames without falling apart.
just walk around with your camera and watch the entire world get rebuilt in 3D at 20 fps.
→ runs at ~20 FPS on a single GPU
→ Stable over 10,000+ frames
→ Beats optimization-based methods on benchmarks
→ Works on drone footage, driving videos, indoor walkthroughs
100% open source.
More footage showing an interceptor trying yet failing to down and the resulting impact of an Iranian ballistic missile earlier in the Southern Israeli city of Dimona.
when you finish rebuilding @karpathy nanochat only to remember your actual job for the foreseeable future is still gonna be building agents that are just OpenAI calls in a for loop
This is interesting as a first large diffusion-based LLM.
Most of the LLMs you've been seeing are ~clones as far as the core modeling approach goes. They're all trained "autoregressively", i.e. predicting tokens from left to right. Diffusion is different - it doesn't go left to right, but all at once. You start with noise and gradually denoise into a token stream.
Most of the image / video generation AI tools actually work this way and use Diffusion, not Autoregression. It's only text (and sometimes audio!) that have resisted. So it's been a bit of a mystery to me and many others why, for some reason, text prefers Autoregression, but images/videos prefer Diffusion. This turns out to be a fairly deep rabbit hole that has to do with the distribution of information and noise and our own perception of them, in these domains. If you look close enough, a lot of interesting connections emerge between the two as well.
All that to say that this model has the potential to be different, and possibly showcase new, unique psychology, or new strengths and weaknesses. I encourage people to try it out!