De-An Huang

Research Scientist @NVIDIAAI, CS PhD @StanfordAILab, Robotics MS @CMU_Robotics

[Wonmin Byeon @wonmin_byeon]

🚀 New paper: STORM — Efficient VLM for Long Video Understanding

STORM cuts compute costs by up to 8× and reduces decoding latency by 2.4–2.9×, while achieving state-of-the-art performance.

Details + paper link in the thread ↓

[Danfei Xu @danfei_xu]

Thrilled to share this story covering our collaboration with @meta_aria @RealityLabs !

Human data is robot data in disguise. Imitation learning is human modeling.

We are at the beginning of something truly revolutionary, both for robotics and human-level AI beyond language.

[Boyi Li @Boyiliee]

I’ve dreamt of creating a tool that could animate anyone with any motion from just ONE image… and now it’s a reality!
🎉 Super excited to introduce updated 3DHM: Synthesizing Moving People with 3D Control.
🕺💃3DHM can generate human videos from a single real or synthetic human image. #Animation #GenAI #AI #3DHM
✨ The magic of 3D control? Turning 2D pixels into lifelike, animated humans.
🎥 Check out our demo (and Merry Christmas)! https://t.co/QlIZxHsj78

Paper: https://t.co/hOIN9q5kKF
Github: https://t.co/7NS9rMw7iM
Webpage: https://t.co/KadBR9XGSU

Proudly working with the great @JunmingChenleo, @jathushan, @YGandelsman, Alyosha Efros and @JitendraMalikCV😃

Kindly note: This video is intended solely for research purposes and is not authorized for commercial use.

[Homanga Bharadhwaj @mangahomanga]

HandsOnVLM: An in-context action prediction assistant for daily activities.

It enables predicting future interaction trajectories of human hands in a scene given natural language queries.

Evaluations across 100s of diverse scenarios in homes, offices, and outdoors!

1/n

[Danfei Xu @danfei_xu]

I gave an Early Career Keynote at CoRL 2024 on Robot Learning from Embodied Human Data

Recording: https://t.co/U2jJ8jYpss

Slides: https://t.co/YNbMJmbth3

Extended summary thread 1/N

[Subhashree Radhakrishnan @subhashree_r]

Hiring interns for our Metropolis Foundation Models Applied Research team at NVIDIA for Spring/ Summer 2025. Internship topics will pertain to 2D/ 3D VLMs and Vision Centric Foundation Models. If you are at ECCV, let's have a chat.

[De-An Huang @deanh_tw]

@tamaybes the link to Claude 🤣

[Pavlo Molchanov @PavloMolchanov]

🚀 Our team is hiring! Join to Advance Efficiency in Deep Learning at NVIDIA! 🚀

🔗 Apply here: https://t.co/aFgGBQKBED

Our team, Deep Learning Efficiency Research (https://t.co/rOIpUX37nP) at NVIDIA Research, is about a year old, and we are expanding. We're looking for talented individuals at all levels, from PhD students in their final year to experienced researchers (the position will be updated accordingly).

Requirements:
✓ PhD (or in the final year of your program)
✓ Multiple tier-one conference papers (ICLR, CVPR, ICML, NeurIPS, etc.)
✓ Strong background in efficient models, transformer architectures, pruning, quantization, NAS, etc.
✓ Experience with Large Language Models (LLMs) and Vision-Language Models (VLMs)

What We Do:
• Explore applications of Large Language Models (LLMs) and Multi-Modal Language Models
• Enhance post-training model optimization via compression, sparsity, quantization, and Neural Architecture Search (NAS)
• Design novel and efficient architectures
• Implement dynamic and adaptive model inference
• Develop innovative training paradigms
• Enhance training data

What to Include in Your Application:
- Google Scholar link
- Recent list of publications
- List ideas you’re eager to explore in efficiency
- Ideally in a short CV in the single PDF file

You can also send an email to pmolchanov [at] https://t.co/PY7acTOSEz with the subject line [HIRING DLER 2024] <your name>.

Join a great team: Saurav Muralidharan, Hongxu (Danny) Yin, Xin Dong, SHIZHE DIAO, Peter Belcak, Greg Heinrich, Yingyan (Celine) Lin

Our members have produced a lot of high-quality work over the past year, including Minitron, VILA, Flextron, X-VILA, DoRA, RADIO, LITA and more.

[Apparate Labs @apparatelabs]

Introducing Proteus 0.1, REAL-TIME video generation that brings life to your AI.

Proteus can laugh, rap, sing, blink, smile, talk, and more. From a single image!

Come meet Proteus on Twitch in real-time. ↓
Sign up for API waitlist: https://t.co/G0BaHDzhXF
1/11

[De-An Huang @deanh_tw]

@_akhaliq Thanks for posting our paper!
The code is available here: https://t.co/Hctb5OEas3

[De-An Huang @deanh_tw]

Most existing Video LLMs struggle with the “When?” questions. In contrast, our proposed LITA can answer challenging "when" questions like “When does the girl show resilience in her performance?”

Paper: https://t.co/hfxH6zIGNd
Code: https://t.co/Hctb5OEas3

[AK @_akhaliq]

LITA

Language Instructed Temporal-Localization Assistant

There has been tremendous progress in multimodal Large Language Models (LLMs). Recent works have extended these models to video input with promising instruction following capabilities. However, an important

[Jan Kautz @jankautz]

#NVIDIA's new Visual Insight Agent (VIA), shown in the #GTC keynote, is collection of workflows to build visual AI agents. You can get rich summaries of activities in natural language or ask detailed questions about events that occurred. Give it a try!
https://t.co/t686AHkziA

[Chaowei Xiao @ChaoweiX]

🚨 Your chat in #openai #ChatGPT could be stolen😱.
#Safety/#security analysis needs to look at the entire system instead of just the #LLM!!!
Welcome to A new era of #LLM #security: Exploring Security Concerns in Real-World LLM-based Systems.
​https://t.co/20G8lngKZw

[Xingyu Lin @Xingyu2017]

What state representation should robots have? 🤖 I’m thrilled to present an Any-point Trajectory Model (ATM), which models physical motions from videos without additional assumptions and shows significant positive transfer from cross-embodiment human and robot videos! 🧵👇

[De-An Huang @deanh_tw]

My favorite example of T-Stitch is Ghibli Diffusion :) Stitching with small model in fact improve prompt alignment (see the NYC background).

[AK @_akhaliq]

🖇 T-Stitch

Accelerating Sampling in Pre-trained Diffusion Models with Trajectory Stitching

Sampling from diffusion probabilistic models (DPMs) is often expensive for high-quality image generation and typically requires many steps with a large model. In this paper, we introduce sampling Trajectory Stitching T-Stitch, a simple yet efficient technique to improve the sampling efficiency with little or no generation degradation. Instead of solely using a large DPM for the entire sampling trajectory, T-Stitch first leverages a smaller DPM in the initial steps as a cheap drop-in replacement of the larger DPM and switches to the larger DPM at a later stage. Our key insight is that different diffusion models learn similar encodings under the same training data distribution and smaller models are capable of generating good global structures in the early steps. Extensive experiments demonstrate that T-Stitch is training-free, generally applicable for different architectures, and complements most existing fast sampling techniques with flexible speed and quality trade-offs. On DiT-XL, for example, 40% of the early timesteps can be safely replaced with a 10x faster DiT-S without performance drop on class-conditional ImageNet generation. We further show that our method can also be used as a drop-in technique to not only accelerate the popular pretrained stable diffusion (SD) models but also improve the prompt alignment of stylized SD models from the public model zoo.

[De-An Huang @deanh_tw]

Check out our new work on speeding up DiT!

[Zizheng Pan @zizhpan]

Both Sora and Stable Diffusion 3 adopt diffusion transformers, but do we really need a super large DiT for all sampling steps for generation?🧐

No🙅‍♂️. We found ~40% early timesteps of DiT-XL can be replaced with a 10x faster DiT-S without image quality drop!

Introduce Trajectory Stitching (T-Stitch), a training-free method that complements existing efficient sampling methods by dynamically allocating computation to different denoising steps.

Paper: https://t.co/mfoGpMkSXH
Code: https://t.co/Grbuxy3qrV
Project page: https://t.co/4CK3ePokQR

[Chulin Xie @ChulinXie]

Our benchmark is on @huggingface leaderboard!!

[Boyi Li @Boyiliee]

Super excited to announce our new work: Synthesizing Moving People with 3D Control (3DHM)💡

Why is 3DHM unique?
With 3D Control, 3DHM can animate a 𝗿𝗮𝗻𝗱𝗼𝗺 human photo with 𝗮𝗻𝘆 poses in a 𝟯𝟲𝟬-𝗱𝗲𝗴𝗿𝗲𝗲 camera view and 𝗮𝗻𝘆 camera azimuths from 𝗮𝗻𝘆 video!

[Arash Vahdat @ArashVahdat]

📢🔥 My team at NVIDIA Research is looking for full-time research scientists &amp; summer interns.

Topics of interest are:
1⃣Gen AI for science (climate, biology, chemistry)
2⃣Image/Video/3D (generate, edit, manipulate)
3⃣Fundamental generative learning

Apply via links below

[Yen-Chia Hsu @yenchiah]

The deadline for my PhD recruitment vacancy is approaching (about two weeks away). Please do apply for it if you are interested in the position!