Chenhao Zheng

Computer Vision PhD student @uwcse | Student Reseacher @allen_ai | ex Undergrad @UMichCSE and @sjtu1896

[Chenhao Zheng @Michael3014018]

We believe in the representation of 3d point trajectory as an alternative form of world modeling. it is universal to all object, more compact than pixel, and can directly be consumed by many downstream task.

p.s. the key for us to generate high quality real world data is we respect objectness and do hard work in object grounding -- which allows us to properly filter and smooth noisy trajectories coming out of off the shell 3d tracking methods.

[Ai2 @allen_ai]

We're releasing MolmoMotion, a 3D motion forecasting model.

Given one or a few video frames, 3D points on an object, & an instruction like "Put the white bowl on the table," MolmoMotion predicts where those points will go over the next few seconds in a shared 3D world frame. 🧵

[DailyPapers @HuggingPapers]

Ai2 just released MolmoMotion on Hugging Face

A 4B vision-language model

that forecasts 3D point trajectories

from video and language instructions,

predicting how objects move in space over time.

[Yinuo Yang @yinuoy01]

Check out our new work, You Only Judge Once: a 4B multimodal reward model that scores all N responses in a single forward pass. It's both faster (up to N×) and more accurate than scoring each in isolation. As a GRPO reward, it also delivers more gains on open-ended generation than single-response RMs.

📄 https://t.co/E43wElF15H
🤖 https://t.co/vsKOHyVNS3
📊 https://t.co/xm1Vsx1hYl
💻 https://t.co/tTuhWywIHC

[Chenhao Zheng @Michael3014018]

@yinuoy01 Amazing work!

[Chenhao Zheng @Michael3014018]

Extremely grateful to work with the amazing team!!
@JieyuZhang20, @mrezasal1, Ziqi Gao, Vishnu Iyengar, Norimasa Kobori, Quan Kong, @RanjayKrishna

[Chenhao Zheng @Michael3014018]

For efficiency,despite the cost of trajectory extraction, TrajViT trains faster, consumes less GPU memory, and performs quicker inference for sequences ≥ 64 frames.

[Chenhao Zheng @Michael3014018]

We also connect TrajViT and ViT3D to Llama-3, forming two VideoLLMs. Across six Video-QA benchmarks, TrajViT-LLM delivers +5.24 pp accuracy, trains 4 × faster* and requires 18 × fewer inference FLOPs than the ViT3D counterpart. We also demonstrate in the paper that our model has a good scaling behavior, in terms of both training data size and inference frame number. Learn more in our paper https://t.co/aewxhdAgI1 or project page: https://t.co/9KbOxwXoSU!

[Chenhao Zheng @Michael3014018]

For evaluation, we compare it with standard ViT with space-time patch tokens (ViT3D) and state-of-the-art token merging methods. Task scope: video-text retrieval, spatiotemporal detection, action classification, and VideoLLM QA. TrajViT surpasses ViT3D on all tasks. At our largest scale, we see +6 pp top-5 recall over ViT3D while carrying only 10 % of its token load.

[Chenhao Zheng @Michael3014018]

Having trouble dealing with the excessive token number when processing a video? Check out our paper that is accepted by ICCV 2025 with an average score of 5.5! We tokenize video with tokens grounded in trajectories of all objects rather than fix-sized patches. Trained with a CLIP objective, TrajViT is the first model to consistently outperform standard ViT with all video understanding tasks we tested even with modern Video-LLM QA—while using ~10 × fewer tokens.
paper: https://t.co/aewxhdAgI1
project page: https://t.co/9KbOxwXoSU
Details in the thread.

[Chenhao Zheng @Michael3014018]

We train a video encoder, TrajViT, using our tokenization paradigm with CLIP objective, on a large scale dataset of 50M image + 10M video. TrajViT can naturally process image data by treating each image segment as a trajectory of length one, allowing seamingless joint training with both videos and images.

[Chenhao Zheng @Michael3014018]

Naively splitting the video tensor into patches is known to introduce memory bottleneck, but is still the de-facto way of tokenizing video due to its strong performance. As shown in figure, We fundamentally transform traditional video tokenization by reorganizing video tokens to correspond to trajectories of all the objects in a video, similar to how our human perceives the visual stream as discovered by cognitive scientists. The result is a token set that is (i) semantically meaningful and temporally non-redundant, (ii) token number positively related to scene complexity, and (iii) robust to lighting and camera motion.

[Jae Sung Park @jjaesungpark]

🔥We are excited to present our work Synthetic Visual Genome (SVG) at #CVPR25 tomorrow!
🕸️ Dense scene graph with diverse relationship types.
🎯 Generate scene graphs with SAM segmentation masks!

🔗Project link: https://t.co/TY57wucRHW
📍 Poster: #32689, Fri 2-4 PM 👇🧵

[Jieyu Zhang @JieyuZhang20]

Calling all #CVPR2025 attendees!

Join us at the SynData4CV Workshop at @CVPR (Jun 11 full day at Grand C2, starting at 9am) to learn more about recent advancements in synthetic data for CV!

Explore more: https://t.co/XADFhiJn8S

[Ainaz Eftekhar @ainaz_eftekhar]

🎉 Excited to introduce "The One RING: a Robotic Indoor Navigation Generalist" – our latest work on achieving cross-embodiment generalization in robot visual navigation! 🤖🌍
RING is a universal navigation policy trained entirely in simulation on diverse, random embodiments at scale. It requires no real-world training data 🚫🤖 and works out of the box on a wide range of robots!
👉 Project page: https://t.co/FTfooOvXtB

[Jia-Bin Huang @jbhuang0604]

The slide is bad, her response to an audience is even worse…

“Maybe there is one, maybe they are common, who knows what. I hope it was an outlier."

[Chenhao Zheng @Michael3014018]

This project is led by @zitong_lan , with collaboration of @zhiwei_zzz, and advised by @mingmin_zhao . Thank for everyone’s great efforts!

[Chenhao Zheng @Michael3014018]

Excited to share our #NeurIPS2024 spotlight: Acoustic Volume Rendering (AVR) for Neural Impulse Response Fields.

AVR greatly improve the state-of-the-art in novel view spatial audio synthesis by introducing acoustic volume rendering. Listen with headphone for example below

[Chenhao Zheng @Michael3014018]

In parallel with AVR, we develop AcoustiX, an acoustic simulation platform that generates more physically accurate signals compared to simulators like soundspace. Soundspace exhibits significant errors in signal phases and arrival times, so we develop a version to solve these.