Fang Jiading

VLMs @Waymo | PhD @TTIC_connect | To uncover Embodied Spatial Intelligence

[Fang Jiading @jiading_fang]

Just realized they indeed released a post about robotics a several days ago: https://t.co/5f83c5jKBR

[Fang Jiading @jiading_fang]

So, Anthropic is building self-replicating robots right now? 😂

[Anand Bhattad @anand_bhattad]

[5/5] Camera conditioning improves performance across ACT, Diffusion Policy, and SmolVLA, in both simulation and the real world.

Key takeaway: Robot policies may spend substantial capacity inferring camera pose from appearance cues. When camera pose is available (or can be estimated), providing it explicitly can significantly improve viewpoint generalization.

Full video below 👇

Paper: https://t.co/9ZdrPajIQE
Project: https://t.co/Wev8XVvIMG
code: https://t.co/jHut7z9ERW

Congratulations to lead author @JiangTianchong and the entire team: Jingtian Ji, Xiangshan Tan, @jiading_fang, @vitorguizilini, @mattrwalter

[Fang Jiading @jiading_fang]

Really glad to see another MLLM conditioned by cameras!

Check our work conditioning robot learning models on cameras (recently received best robot learning finalist award in ICRA 2026): https://t.co/LRXMyUz8mF

Conditioning on 3D signals is not against end-to-end approaches, instead it exposes the importance of physics grounding in MLLMs, especially when they are relatively cheap and reliable to obtain. I hope it inspires the development of next-gen models that truly understands the physical world.

[Saining Xie @sainingxie]

📸latest in our cambrian series: cambrian-p, p for pose.
i think pose is probably the minimal sufficient 3d signal (and it’s easy to get!) that we need for robust video multimodal models -- jointly modeling frames and pose turns image sequences into a globally grounded structure.

[Fang Jiading @jiading_fang]

Glad this shell game task gets more attention from @GeneralistAI and @RhodaAI for measuring long-context robot manipulation capabilities. We explored this robotic task 3 years ago with state-based LLMs. Check out Statler: https://t.co/gq7hdpDa2W.

[Generalist @GeneralistAI]

GEN-1 plays the 🐚 shell game, trained on just 1 hr of robot data. It also generalizes to unseen objects, like @BerkayAntmen 's car keys.

Physical AI models should be capable of benchmark tasks like this one. It's interesting for the all the reasons @RhodaAI calls out -- requires visual memory, and the model must track the cups from the very start, at high frame rates.

Interestingly, GEN-1 appears to exhibit a degree of "active perception." It's subtle; the hands can sometimes appear to "follow" the cups, using its own movements to help attend to where it thinks the object should be.

Read more about GEN-1 in our blog post in the comments below ↓

[Fang Jiading @jiading_fang]

This is a quite important question because it’s about how to enforce HARD constraints, the very problem people facing trying to use world models for physics simulation. Many comments mention data + RL. I get the data part; and for the RL part, it’s possible to do OCR or finger detection, but how to RL physics?

[Lucas Beyer (bl16) @giffmana]

I have a question about last year's image-generation progress, wonder what y'all think.

How did we go from all models consistently getting fingers wrong, to all models consistently getting them right?

This "flip" seems to have happened basically across all companies/models at the ~same time.

Even "random" non-frontier papers seem to get it right? Or they just cherry-pick the figures?

[Fang Jiading @jiading_fang]

For Vincent to come out and cannibalize his very own contributions to the field—3D and scene reps—is an absolutely courageous and chivalrous move. It makes this piece painfully credible. Real-world intelligence is the only lead.

[Vincent Sitzmann @vincesitzmann]

In my recent blog post, I argue that "vision" is only well-defined as part of perception-action loops, and that the conventional view of computer vision - mapping imagery to intermediate representations (3D, flow, segmentation...) is about to go away.

https://t.co/aFmE9CHHau

[Waymo @Waymo]

We’re excited to introduce the Waymo World Model—a frontier generative mode for large-scale, hyper-realistic autonomous driving simulation built on @GoogleDeepMind’s Genie 3.

By simulating the “impossible”, we proactively prepare the Waymo Driver for some of the most rare and complex scenarios—from tornadoes to planes landing on freeways—long before it encounters them in the real world.

https://t.co/EbMut47ZEY

[Fang Jiading @jiading_fang]

Presenting tomorrow (10/21 Tuesday) afternoon at poster session 2 #359.

Let’s discuss about reconstructing articulated realistic environments for robot simulations🤖!
#ICCV2025

[Fang Jiading @jiading_fang]

Ever want to reconstruct and animate everyday articulated objects with no 3D scans or category priors?

🚀Introducing SplArt: Articulation Estimation & Part-Level Reconstruction with 3D Gaussian Splatting!
#3Dvision #GaussianSplatting

[Fang Jiading @jiading_fang]

Great work Ankit! Feels the bitter-lesson all-over again smh. Retaining the original vocabulary is probably a big part, although it may be hard to properly ablate.

I used to not believe that VLMs can do geometric or numerical works better than vision/action-specific models, but more and more evidences are proving me wrong.

[Fang Jiading @jiading_fang]

This is the last piece of work from my PhD (from almost one year ago). And I'm proud of the solid study we have done.

Huge shoutout to the project lead @JiangTianchong , and colleagues Jingtian Ji, Xiangshan Tan to make my wild thought a reality. They did excellent work in implementation and uncovered many crucial tricks to enable the model.

Also huge thanks to all the collaborators @anand_bhattad, Vitor Guizilini and Matthew Walter for continuous support and guidance along the way.

Please let us know if you have any questions, this is only a first step towards answering a grand quest.

[Fang Jiading @jiading_fang]

🤖Ever wonder why robotic IL models/VLAs never make use of the camera info (extrinsics & intrinsics)?

We wonder the same. So we created ways to inject camera info and carefully studied its impacts. In fact, it's important to "Know Where Your Camera Is"!

https://t.co/gK0SbSW1km

[Fang Jiading @jiading_fang]

The idea originated from the observation that, at the time, almost none of the imitation learning or VLA models take into input the camera parameters like extrinsics and intrinsics, but just using RGB frames. Often the times, people don't even bother to properly calibrate the camera, hoping that large-scale training can solve it all.

While such dream is not impossible from a scaling perspective, one has to consider the data scarcity and efficiency in reality. From the 3D community, we know it's difficult to reconstruct the scene if the camera parameters are off. If we assume 3D understanding is the pre-requisite of learning robot motion, the current approaches for VLAs are basically trying to solve a harder task with even less data. It creates a big gap in our understanding.

Thus we set up to study the element of camera inside VLA or imitation learning models.

[François Fleuret @francoisfleuret]

"the human brain doesn't need tons of training data to do stuff"

[Fang Jiading @jiading_fang]

@ssahoo_ “Romantic” is one of the best descriptions for a PhD life.

[Fang Jiading @jiading_fang]

Interesting paper proving that fixed-size single-vector embedding is not enough for retrieval tasks because the dimension required can be arbitrarily high (up to the size of the system itself where the vector provides no compression). Its implications go beyond retrieval.

While the paper mainly considers the retrieval tasks, the results apply to other domains using similar techniques. On top of my head, visual-text embedding like CLIP or CoCA; VQ-VAE methods.

The usages of CLIP or CoCA are mainly
a) for contrastive learning between language and image embeddings, such as zero-shot classification. The result directly applies, meaning no dimension is high enough, and there are going to be images that are misclassified.

b) for image embedding that combines with language tokens to be decoded by language decoders. This is similar to the "cross-encode" exception where the query text and the retrieval are encoded together and can achieve 100% accuracy on the LIMIT data using Gemini-2.5-pro.

VQ-VAE is another popular "compression" method used for image reconstruction and generation, where the representation is a set of vectors from a fixed dictionary. In this case, the representation belongs to the "multi-vector" category, and is not cursed by the results.

[Orion Weller @orionweller]

Instructions/reasoning are now everywhere in retrieval - we want embeddings to do it all! 🚀

But... is it even possible? 🤔

Turns out, it's not possible for single-vector models 😱 theoretically and empirically! To make it obvious we OSS a simple eval SoTA models flop on!

🧵