Jisu Nam

Intern @Adobe (2x) @NaverCloud | Ph.D @KAIST

[Wildminder @wildmindai]

Wan 2.1 + LoRA for dense 3D tracking
TrackCraft3R tracks every pixel and actually gets 3D motion right

- fast, accurate
- predicts visibility maps and 3D residuals

https://t.co/WwNffdCsmN

[Kwang Moo Yi @kwangmoo_yi]

Nam et al., "TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking"

Lora-finetune a video model to take in dense point map estimates of each frame, with proper alignment and encoding == significant 3D tracking improvement.

[AIQUEST @AiquestAcademy]

Video DiTs aren't just for generation. They are secretly incredible physics engines.

TrackCraft3R repurposes a pretrained Video DiT into a dense 3D tracker. It uses internet-scale motion priors to track every pixel in a monocular video in a single forward pass.

Results:
* New SOTA 3D tracking
* 1.3x faster
* 4.6x less peak memory

Project demo page is available.
#ComputerVision #AI

[Wildminder @wildmindai]

CS:GO + Adobe + Wan2.1 = WorldCam.
Interactive autoregressive 3D gaming worlds.

> AI now generates playable 3D worlds live
> You move the mouse, the AI builds the map instantly
> Turn around, and it remembers exactly what was there

No traditional game engines. Just neural networks hallucinating reality.

https://t.co/2WWhVacW7D

[AK @_akhaliq]

WorldCam

Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation

paper: https://t.co/2Ek7HQSM2d

[Kwang Moo Yi @kwangmoo_yi]

Preprint of today: Nam et al., "Emergent Temporal Correspondences from Video Diffusion Transformers" -- https://t.co/OHwAsFwYXo

Not as good as other point trackers, but Video DiT's themselves can generate point tracks as is. This can further be used to enhance generation.

[Jisu Nam @jisu__nam]

DiffTrack: Emergent Temporal Correspondences from Video Diffusion Transformers (KAIST & GoogleDeepMind)

How do Video Diffusion Transformers (Video DiTs) generate motion-consistent videos? How do they internally establish and represent temporal correspondences across frames?

We answer this fundamental question with DiffTrack — an analysis framework for uncovering and exploiting emergent temporal correspondences in video diffusion.

What’s inside DiffTrack?

- Quantifies temporal correspondence within DiTs during video generation using novel evaluation metrics.
- Provides a detailed analysis of the open-source video DiT models including CogVideoX, HunyuanVideo, and CogVideoX-I2V.
- Achieves state-of-the-art performance in zero-shot point tracking compared to existing vision foundation and self-supervised video models.
- Improves motion consistency of generated videos with a novel guidance method — without auxiliary models or supervision.

- Project Page: https://t.co/gqQ5lAaDWD
- Paper: https://t.co/G1FPmJaUU7
- Code: https://t.co/KRba7dXAUU

[Naga/Abhi @NagaSaiAbhinay]

DiffTrack enables
1) zero-shot point tracking (points given for first frame, both real world and generated videos)
2) motion-enhanced video generation with a novel guidance method.

[Tom Yeh @ProfTomYeh]

SORA by Hand ✍️

OpenAI’s #SORA took over the Internet when it was announced earlier this year. The technology behind Sora is the Diffusion Transformer (DiT) developed by William Peebles and Shining Xie.

How does DiT work?

𝗚𝗼𝗮𝗹: Generate a video conditioned by a text prompt and a series of diffusion steps

[1] Given
↳ Video
↳ Prompt: "sora is sky"
↳ Diffusion step: t = 3

[2] Video → Patches
↳ Divide all pixels in all frames into 4 spacetime patches

[3] Visual Encoder: Pixels 🟨  → Latent 🟩
↳ Multiply the patches with weights and biases, followed by ReLU
↳ The result is a latent feature vector per patch
↳ The purpose is dimension reduction from 4 (2x2x1) to 2 (2x1).
↳ In the paper, the reduction is 196,608 (256x256x3)→ 4096 (32x32x4)

[4] ⬛ Add Noise
↳ Sample a noise according to the diffusion time step t. Typically, the larger the t, the smaller the noise.
↳ Add the Sampled Noise to latent features to obtain Noised Latent.
↳ The goal is to purposely add noise to a video and ask the model to guess what that noise is.
↳ This is analogous to training a language model by purposely deleting a word in a sentence and ask the model to guess what the deleted word was.

[5-7] 🟪 Conditioning by Adaptive Layer Norm

[5] Encode Conditions
↳ Encode "sora is sky" into a text embedding vector [0,1,-1].
↳ Encode t = 3 to as a binary vector [1,1].
↳ Concatenate the two vectors in to a 5D column vector.

[6] Estimate Scale/Shift
↳ Multiply the combined vector with weights and biases
↳ The goal is to estimate the scale [2,-1] and shift [-1,5].
↳ Copy the result to (X) and (+)

[7] Apply Scale/Sift
↳ Scale the noised latent by [2,-1]
↳ Shifted the scaled noised latent by [-1, 5]
↳ The result is "conditioned" noise latent.

[8-10] Transformer

[8] Self-Attention
↳ Feed the conditioned noised latent to Query-Key function to obtain a self-attention matrix
↳ Value is omitted for simplicity

[9] Attention Pooling
↳ Multiply the conditioned noised latent with the self-attention matrix
↳ The result are attention weighted features

[10] Pointwise Feed Forward Network
↳ Multiply the attention weighted features with weights and biases
↳ The result is the Predicted Noise

🏋️‍♂️ 𝗧𝗿𝗮𝗶𝗻

[11]
↳ Calculate MSE loss gradients by taking the different between the Predicted Noise and the Sampled Noise (ground truth).
↳ Use the loss gradients to kick off backpropagation to update all learnable parameters (red borders)
↳ Note the visual encoder and decoder's parameters are frozen (blue borders)

🎨 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗲 (𝗦𝗮𝗺𝗽𝗹𝗲)

[12] Denoise
↳ Subtract the predicted noise from the noised latent to obtain the noise-free latent

[13] Visual Decoder: Latent 🟩 → Pixels 🟨
↳ Multiply the patches with weights and biases, followed by ReLU

[14] Patches → Video
↳ Rearrange patches into a sequence of video frames.