Tao

PhD student @Cornell_CS.

[Žiga Kovačič @zzigakovacic]

Do coding agents understand the dynamics of the world well enough to reconstruct it from video?

We introduce 4DCodeBench to evaluate this ability through 4D inverse graphics.

https://t.co/QlkzTPZpbi
<🧵>

[Yen-Chen Lin @yen_chen_lin]

Video generation models exploded onto the scene in 2024, sparked by the release of Sora from OpenAI. I wrote a blog post on key techniques that are used in building large video generation models:

https://t.co/fsqdefXdhr

[Gengshan Yang @gengshanY]

Sharing my recent project, agent-to-sim:

From monocular videos taken over a long time horizon (e.g., 1 month), we learn an interactive behavior model of an agent (e.g., a 🐱) grounded in 3D.
https://t.co/Q1aqledjys

[Tao @ttao_tw]

Molmo is amazing BUT a bad reader when it comes to mileage signs🧐
@mattdeitke @allen_ai

[Ai2 @allen_ai]

Meet Molmo: a family of open, state-of-the-art multimodal AI models.

Our best model outperforms proprietary systems, using 1000x less data.

Molmo doesn't just understand multimodal data—it acts on it, enabling rich interactions in both the physical and virtual worlds.

Try it for yourself: https://t.co/IWKKUdfxlg

[Guy Tevet @GuyTvt]

[1/4] Can we sample 3D from a 2D diffusion models instead of optimizing with SDS?
In "MAS: Multi-view Ancestral Sampling for 3D motion generation using 2D diffusion" [CVPR2024🥳] we show that for 3D animation, the answer is - Yes!

[OpenAI @OpenAI]

Introducing Sora, our text-to-video model.

Sora can create videos of up to 60 seconds featuring highly detailed scenes, complex camera motion, and multiple characters with vibrant emotions.

https://t.co/YYpOAcrXQ3

Prompt: “Beautiful, snowy Tokyo city is bustling. The camera moves through the bustling city street, following several people enjoying the beautiful snowy weather and shopping at nearby stalls. Gorgeous sakura petals are flying through the wind along with snowflakes.”

[Aleksander Holynski @holynski_]

Check out our new paper that turns a (single image) =&gt; (interactive dynamic scene)!

I’ve had so much fun playing around with this demo.

Try it out yourself on the website: https://t.co/zQfIIEKrmU

[Michael Black @Michael_J_Black]

I get a lot of reviews that say my work is not novel and I bet I'm not alone. It's always frustrating because I see novelty where the reviewer doesn't. Rather than rebut every critique, I've written a blog post to help reviewers think about novelty. https://t.co/UXLabOkYcn

[Tao @ttao_tw]

@Yu_Ju_Tsai @iclr_conf Congrats!

[AK @_akhaliq]

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation

paper page: https://t.co/QYyAXoAZj6

Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

[Matthias Niessner @MattNiessner]

Check out 𝐌𝐨𝐭𝐢𝐨𝐧𝟐𝐕𝐞𝐜𝐒𝐞𝐭𝐬, a 4D diffusion model for dynamic surface reconstruction from imperfect observations of sparse, noisy, or partial point clouds.

Main idea: we represent time-varying shapes via 4D neural representation with latent vector sets, and then explicitly learns the shape and motion distribution of non-rigid objects through an iterative denoising process of latent vector sets.

https://t.co/rDHhs5lcRT
https://t.co/7jELNZIoqu
https://t.co/B4Bil0vYzq

Great work by @vveicao @changlias @_BiaoZhang @jiapeng_tang

[AK @_akhaliq]

Learning the 3D Fauna of the Web

paper page: https://t.co/PoGxGOsJj6

Learning 3D models of all animals on the Earth requires massively scaling up existing solutions. With this ultimate goal in mind, we develop 3D-Fauna, an approach that learns a pan-category deformable 3D animal model for more than 100 animal species jointly. One crucial bottleneck of modeling animals is the limited availability of training data, which we overcome by simply learning from 2D Internet images. We show that prior category-specific attempts fail to generalize to rare species with limited training images. We address this challenge by introducing the Semantic Bank of Skinned Models (SBSM), which automatically discovers a small set of base animal shapes by combining geometric inductive priors with semantic knowledge implicitly captured by an off-the-shelf self-supervised feature extractor. To train such a model, we also contribute a new large-scale dataset of diverse animal species. At inference time, given a single image of any quadruped animal, our model reconstructs an articulated 3D mesh in a feed-forward fashion within seconds.

[Zhiyang (Frank) Dou @frankzydou]

🔥Yes! You can achieve REAL-TIME text-to-motion generation using a simulated humanoid to perform various skills!
This feat is realized through the integration of PHC and EMDM.
💬This combination addresses two pivotal challenges in human motion synthesis: ensuring physical plausibility and enhancing motion generation efficiency.

Hats off to @zhengyiluo for the fantastic work! ☺️

PHC: https://t.co/Kx4BM1xpCE
EMDM: https://t.co/pGTAduj1D9

#AIGC #Animation #Character #Generation #3D #AI #Motion #SMPL #Graphics #SOSMPL #IsaacGym #Simulation

[Chieh Hubert Lin @ChiehHubertLin]

(1/5) We just released our new paper Virtual Pets!

Recent generative models are getting us a bit closer to a fully synthetic virtual environment. BUT!! Isn't it too boring without a cat😾? Here we go!

Project: https://t.co/mqeSajjfiM
ArXiv: https://t.co/ObjBnXyvX3

[Karsten Kreis @karsten_kreis]

📢📢 Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models

https://t.co/MZIXSQlq1q

We generate dynamic 4D assets and scenes with score distillation!

w/ the amazing @HuanLing6*, @seungkim0123*, Antonio Torralba, @FidlerSanja

(1/n)

[Michael Black @Michael_J_Black]

WHAM defines the new state of the art in 3D human pose estimation from video. By a large margin. It’s fast, accurate, and it computes human pose in world coordinates. It’s also the first video-based method to be more accurate than single-image methods. 1/8

[Zian Wang @zianwang97]

🚀 Introducing our #SIGGRAPHAsia work “Adaptive Shells”, a novel #NeRF formulation that yields high visual fidelity and greatly accelerates rendering.

TLDR: Auto-derived bounding shells result in up to 10x faster inference than InstantNGP!

[1/n]

[Agrim Gupta @agrimgupta92]

We introduce W.A.L.T, a diffusion model for photorealistic video generation. Our model is a transformer trained on image and video generation in a shared latent space. 🧵👇

[Tao @ttao_tw]

Big shoutout to @ChiehHubertLin , my incredible mentor in this project, for featuring our #DreaMo.
Special thanks to @_akhaliq for reposting our work.

[Chieh Hubert Lin @ChiehHubertLin]

We just released our new paper "DreaMo: Articulated 3D Reconstruction From A Single Casual Video"

With a *single* video, DreaMo reconstructs the 3D shape and discovers the articulation for easy posing.

Video intro: https://t.co/lhdmSKeIak
Project: https://t.co/vGKnwdS05C

[Tao @ttao_tw]

@izac_d_r @ChiehHubertLin I have updated the link. It should work now!