Dongyu Gao

[Yan Chang @Dr_YanChang]

🚀 NVIDIA Isaac Video → Data is now open source—enabling a Real → Sim → Real workflow for robot learning.
The pipeline transforms real-world human demonstrations into simulation-ready assets and training data, then brings learned policies back to physical robots.
🎥 Segment long videos and retrieve relevant action clips with the Video Ingestion Agent
🧩 Reconstruct hand/body-object motion, depth, meshes, and 6-DoF trajectories
🤖 Retarget human motion to robot embodiments and train RL policies in Isaac Lab
🦾 Scale dexterous manipulation with using Contact
Wrench Guidance From Human Demonstration (CHORD)
📚 Access a 1,215-task motion library and 121,500 robot episodes
Real demonstrations → simulation → learned policies → real robots.
Code: https://t.co/erEhNoq49k
Documentation: https://t.co/R9P3s4aXmc
Dataset: https://t.co/C0j9tzYQEw
CHORD: https://t.co/DrHoY4HHtx
A huge thank-you to the amazing contributors whose work made this release possible.
We welcome your feedback and open-source contributions. Join us in building the future of robot learning!
#NVIDIA #Isaac #PhysicalAI #Robotics #RobotLearning #OpenSource

[Chester @chesterzelaya]

The Problem of Dynamic Scenes for SLAM

Traditional SLAM relies on one core assumption -- the scene can not be moving. This makes the math work. The stack of algorithms are created to be able to track *stationary* landmarks in a set of images, and reconstruct its motion

When things move in the image, the reconstructed movement has fatal cascading effects that lead to unusable data

Learning based SLAM is a growing sub-specialization tackling this issue. Its main approach to handling dynamic scenes is to leverage uncertainty throughout the stack of algorithms. This would, for example, prevent pixels of a human moving in frame to be treated the same as the stationary building in the background

Watching legendary algorithms get rejuvenated with our advancements in uncertainty is a breath of fresh air

[Yotam @yotam_erel]

🧵 PractiLight: Practical Light Control Using Foundational Diffusion Models
🔗 https://t.co/YweM7ir948

Diffusion models already “know” a lot about light transport. No need for massive finetuning that hurts generalization. {1/6} 👇

[Hanna Yukhymenko @a_yukh]

Releasing the Jupyter Agent Dataset! 🚀

Training on this data dramatically improves the ability to execute code and analyze data.

Built from 7 TB of real Kaggle datasets + 20k notebooks, creating real code exec traces using Qwen3-Coder and E2B.

https://t.co/mBBjeMGvQy

[Aleksa Gordić (水平问题) @gordic_aleksa]

New in-depth blog post - "Inside vLLM: Anatomy of a High-Throughput LLM Inference System". Probably the most in depth explanation of how LLM inference engines and vLLM in particular work!

Took me a while to get this level of understanding of the codebase and then to write up this one - i quickly realized i understimated the effort. 😅 It could have easily been a book/booklet (lol).

I covered:

* Basics of inference engine flow (input/output request processing, scheduling, paged attention, continuous batching)

* "Advanced" stuff: chunked prefill, prefix caching, guided decoding (grammar-constrained FSM), speculative decoding, disaggregated P/D

* Scaling up: going from smaller LMs that can be hosted on a single GPU all the way to trillion+ params (via TP/PP/SP) -> multi-GPU, multi-node setup

* Serving the model on the web: going from offline deployment to multiple API servers, load balancing, DP coordinator, multiple engines setup :)

* Measuring perf of inference systems (latency (ttft, itl, e2e, tpot), throughput) and GPU perf roofline model

Lots of examples, lots of visuals!

---

I realize i've been silent on social - many of you noticed and thanks for reaching out! :) --> I'm so back! lots of things happened.

Also, in general, I'm a bit sick of superficial content, it really is an equivalent of junk food (h/t @karpathy).

I want to do the best/deepest technical work of my life over the next years and write much more in depth (high quality organic food ;)) so I might not be as frequent around here as i used to be (? we'll see). I'll make it a goal to share a few paper summaries a week or stuff that's relevant / in the zeitgeist.

If you have any topics that happened over the past few weeks/months drop it down in the comments i might focus on some of those in my next posts.

---

Huge thank you to @Hyperstackcloud for giving me an H100 node to run some of the experiments and analysis that i needed to write this up. The team there led by Christopher Starkey is amazing!

Also a big thank you to Nick Hill (who did a very thorough review of the post - basically a code review lol; Nick's a core vLLM contributor and principal SWE at RedHat) and to my friends Kyle Krannen (NVIDIA Dynamo), @marksaroufim (PyTorch), and @ashVaswani (goat) for taking the time during weekend when they didn't have to!

[Dongyu Gao @dy24775307]

@wandb @OpenAI OSS Power

[Hunter Lott @Hunter_Lott_]

The Google Gemini team just quietly shipped a new feature, and it's VERY impressive.

STORYBOOK

This new feature allows you to generate your own read-along story book in a matter of minutes!

I just tried this with our Fable the Fox project.

The output produces up to 10 pages and lets you choose one of two voices for narration!

Unfortunately, it doesn't allow sharing yet with enterprise/business accounts, but hopefully that will change soon.

Give it a try and let us know what you think.

Here are some images of the output we created:

[Jia-Bin Huang @jbhuang0604]

Explaining Flow Matching in 4 minutes

[Abe Davis @AbeDavis]

Excited to share our ACM TOG/ SIGGRAPH '25 work on Noise-Coded Illumination (NCI)! We use coded noise to add an invisible watermark to lighting that helps detect fake or manipulated video. With @gfxptr and collaborators @zekun_hao and @SergeBelongie. url: https://t.co/7v8Ilvl4c6

[Sebastian Loeschcke @SLoeschcke]

Excited for @icmlconf in Vienna!
DM me if you want to chat.
I’ll be presenting:
-LoQT: Low Rank Adapters for Quantized Pre-Training (𝐎𝐑𝐀𝐋) - 27 Jul, 10:10-10.30, Hall A1
-Coarse-To-Fine Tensor Trains for Compact Visual Representations - 25 Jul, 11:30 -13.00, Hall C 4-9 #203

[Srishti @_srishtiyadav]

Checkout our latest 📰 "Revealing Values and Opinions in Large Language Models" where we investigate values and opinions across many LLMs for various political propositions ⬇️

https://t.co/2DIjTmhQ1s

w/ @IAugenstein @SergeBelongie @dustin_wright37 @rnav_arora @NadavBorenstein

[Jiaang Li @jiaangli]

Do Vision and Language Models Share Concepts? 🤔👀🧠

We present an empirical evaluation and find that language models partially converge towards representations isomorphic to those of vision models.

📝: https://t.co/y6UDT9O0MP
🧑‍💻: https://t.co/Tm6dVKxaY5

🧵(1/8)

[Vishal Nedungadi @vishalned_]

🌍 I am excited to announce the release of our research project: “MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learning.” 🚀

Check out our preprint and the project page for details on the dataset and our proposed model: https://t.co/gYeqzhdeGI

[Hieu Pham @hyhieu226]

https://t.co/FEcvI56izD

If you learn CUDA, you probably have read the legendary matrix transpose tutorial by Mark Harris.

📚Today, I present to you a CUDA tutorial by friends at @colfaxintl and myself. We explore many CUDA memory concepts like in Mark's tutorial, and more.

Crucially, we show how to implement these concepts in @nvidia's CuTe library. Yes, it's the one that powers Flash Attention⚡️.

I am grateful for friends at @colfaxintl, especially Jay Shah, for inviting me to help them this project.

This collaboration started in December 2023, when I got interested in @nvidia's CuTe library by reading the FlashAttention repo.

Despite my enthusiasm for the the library, learning it was a headache. I literally made no progress after weeks reading CuTe tutorial and code.

Then one day, I stumbled upon Jay's Flash Attention tutorial (https://t.co/oSL36ySSmB). His writing revealed to me the mathematical elegance in CuTe's design.

Teaser: can you imagine abstract algebra concepts -- like integral modules -- have anything to do with efficient deep learning kernels?!

But I digressed. Since reading Jay's works, I have held so much respect for him that being able to collaborate with his team feels profoundly rewarding.

Anyhow, I hope this CUDA tutorial makes you like CUDA programming and the CuTe library a little bit more.

Please stay tuned. We will be back with more tutorials.

[Zhaochong An @ ECCV✈️ @ZhaochongAn]

🚀 Excited to announce our paper accepted at #CVPR2024!

Understanding our 3D world is paramount. Ever wondered how well few-shot segmentation works in 3D point clouds? 🌟

Check out "Rethinking Few-shot 3D Point Cloud Semantic Segmentation" (https://t.co/5GU6rbwvwR)! 🎉

🧵1/n