Tianxing Wu

Ph.D. student at MMLab@NTU

[Tianxing Wu @_tianxing]

Interested in improving the inference quality of diffusion models? Checkout our #FreeInit poster at No.226 from 10:30am to 12:30pm tomorrow (Tuesday 1 Oct). Will be happy to discuss video diffusion models, noise initialization and more! #ECCV2024

[Ziwei Liu @liuziwei7]

📢Another Free Lunch for GenAI📢

We propose #FreeInit, a sampling strategy to improve temporal consistency of video generation at inference time, requiring no training and can be plugged into any diffusion model

- Project: https://t.co/6KNCrs5C8p
- Code: https://t.co/olvF6oiYs0

[Tianxing Wu @_tianxing]

@RaphLeclercAI Thanks for your interest, looking forward to chat with you!

[Tianxing Wu @_tianxing]

Interested in improving the inference quality of diffusion models? Checkout our #FreeInit poster at No.226 from 10:30am to 12:30pm tomorrow (Tuesday 1 Oct). Will be happy to discuss video diffusion models, noise initialization and more! #ECCV2024

[Ziwei Liu @liuziwei7]

📢Another Free Lunch for GenAI📢

We propose #FreeInit, a sampling strategy to improve temporal consistency of video generation at inference time, requiring no training and can be plugged into any diffusion model

- Project: https://t.co/6KNCrs5C8p
- Code: https://t.co/olvF6oiYs0

[Jingkang (Jake) Yang @JingkangY]

Carry your cool backpack 🎒 and come to visit us in the October 1 and 2 afternoon sessions!

Octopus 🐙
- a vision-language model for robots 🤖 or GTA/Minecraft game play 🎮
- We solve it via training the VLM into a good programmer with sight. It calls correct functions and executes.

FunQA 💃🪩🕺
- a testbed to see whether video language models can understand stupid 🤪funny🔥 viral videos.
- Let’s see how your VLMs understand these fun videos!

[Ziwei Liu @liuziwei7]

📢#ECCV24 Welcome to check out our multimodal GenAI work @eccvconf📢

* Generative Models:

- LGM: https://t.co/Z0TQBtWr18
- Large Motion Model: https://t.co/onXthchEOC
- FreeInit: https://t.co/6KNCrs5C8p
- StructLDM: https://t.co/DfNsppARIb
- Comboverse: https://t.co/eDaFnRUA4M

[Tianxing Wu @_tianxing]

Our project page: https://t.co/n8J2iq3O0d
[3/3]

[Tianxing Wu @_tianxing]

FreeInit is accepted to #ECCV2024!🎉Big thanks to co-authors @scy994 @Jiang_Yuming @ziqi_huang_ @liuziwei7 for their efforts!
Also excited to see our explorations on initial noise being adopted in various tasks, e.g. #ConsistI2V(I2V) #VideoElevator(T2V) #FreeTraj(control).
[1/3]

[AK @_akhaliq]

FreeInit: Bridging Initialization Gap in Video Diffusion Models

paper page: https://t.co/RhCifC8bnd

Though diffusion-based video generation has witnessed rapid progress, the inference results of existing models still exhibit unsatisfactory temporal consistency and unnatural dynamics. In this paper, we delve deep into the noise initialization of video diffusion models, and discover an implicit training-inference gap that attributes to the unsatisfactory inference quality. Our key findings are: 1) the spatial-temporal frequency distribution of the initial latent at inference is intrinsically different from that for training, and 2) the denoising process is significantly influenced by the low-frequency components of the initial noise. Motivated by these observations, we propose a concise yet effective inference sampling strategy, FreeInit, which significantly improves temporal consistency of videos generated by diffusion models. Through iteratively refining the spatial-temporal low-frequency components of the initial latent during inference, FreeInit is able to compensate the initialization gap between training and inference, thus effectively improving the subject appearance and temporal consistency of generation results. Extensive experiments demonstrate that FreeInit consistently enhances the generation results of various text-to-video generation models without additional training.

[Tianxing Wu @_tianxing]

We also find this strategy has broader applications. For example in T2I, FreeInit can help SDXL generate very dark/bright images with better text alignment (see below). Further support for #VideoCrafter2 and #SDXL will be updated soon.
[2/3]

[Ziwei Liu @liuziwei7]

🔥Video Generation with Image Prompts🔥

#CVPR2024 We propose *video generation with image prompts* 📽️VideoBooth📽️, providing more direct content control beyond text prompts @CVPR

- Project: https://t.co/V2GxUoYm9J
- Paper: https://t.co/2LxclfWWRF
- Code: https://t.co/F3drBPIW03

[Yuekun Dai @YuekunDai]

And don't forget to check our new dataset! #CVPR2024

[Tianxing Wu @_tianxing]

@ptsi Thanks :)

[AK @_akhaliq]

FreeInit : Bridging Initialization Gap in Video Diffusion Models @Gradio demo is out on @huggingface

demo: https://t.co/4lHfjgdbNV

run with docker: https://t.co/ZdMJ3kbKqZ

duplicate space with private gpu and no queue: https://t.co/GhrFGeqxOb

[Tianxing Wu @_tianxing]

The @Gradio demo for #FreeInit is now out on @huggingface🤗! Feel free to try it out and play around with the parameters :)
HF demo link: https://t.co/ePaHDjMsyA

[AK @_akhaliq]

FreeInit: Bridging Initialization Gap in Video Diffusion Models

paper page: https://t.co/RhCifC8bnd

Though diffusion-based video generation has witnessed rapid progress, the inference results of existing models still exhibit unsatisfactory temporal consistency and unnatural dynamics. In this paper, we delve deep into the noise initialization of video diffusion models, and discover an implicit training-inference gap that attributes to the unsatisfactory inference quality. Our key findings are: 1) the spatial-temporal frequency distribution of the initial latent at inference is intrinsically different from that for training, and 2) the denoising process is significantly influenced by the low-frequency components of the initial noise. Motivated by these observations, we propose a concise yet effective inference sampling strategy, FreeInit, which significantly improves temporal consistency of videos generated by diffusion models. Through iteratively refining the spatial-temporal low-frequency components of the initial latent during inference, FreeInit is able to compensate the initialization gap between training and inference, thus effectively improving the subject appearance and temporal consistency of generation results. Extensive experiments demonstrate that FreeInit consistently enhances the generation results of various text-to-video generation models without additional training.

[Tianxing Wu @_tianxing]

HuggingFace Demo:
https://t.co/ePaHDjMsyA

[Tianxing Wu @_tianxing]

Thanks @_akhaliq for sharing! 🎉

We propose #FreeInit to bridge the training/inference gap of video diffusion models, improving temporal consistency without finetuning.

Project: https://t.co/n8J2iq3O0d
Code: https://t.co/n2wZZ83eU1
Video: https://t.co/YWtceq2eJ9

[AK @_akhaliq]

FreeInit: Bridging Initialization Gap in Video Diffusion Models

paper page: https://t.co/RhCifC8bnd

Though diffusion-based video generation has witnessed rapid progress, the inference results of existing models still exhibit unsatisfactory temporal consistency and unnatural dynamics. In this paper, we delve deep into the noise initialization of video diffusion models, and discover an implicit training-inference gap that attributes to the unsatisfactory inference quality. Our key findings are: 1) the spatial-temporal frequency distribution of the initial latent at inference is intrinsically different from that for training, and 2) the denoising process is significantly influenced by the low-frequency components of the initial noise. Motivated by these observations, we propose a concise yet effective inference sampling strategy, FreeInit, which significantly improves temporal consistency of videos generated by diffusion models. Through iteratively refining the spatial-temporal low-frequency components of the initial latent during inference, FreeInit is able to compensate the initialization gap between training and inference, thus effectively improving the subject appearance and temporal consistency of generation results. Extensive experiments demonstrate that FreeInit consistently enhances the generation results of various text-to-video generation models without additional training.

[Koichi Hamada @hamadakoichi]

Anime Synthesis with FreeInit + AnimateDiff
#FreeInit #AnimateDiff #StableDiffusion #AIAnime #AIArt

[Tianxing Wu @_tianxing]

@DigThatData @_akhaliq Yeah it looks like this partial renoising strategy may have some similar functionalities. Very interesting. Thanks for discussion! I think the training-inference gap is the core problem, and other better approaches can be developed to address this.

[Tianxing Wu @_tianxing]

@DigThatData @_akhaliq In the ablation, we show that using multiple passes w/o NR can indeed improve consistency, but not as good. From my experience, multiple passes cannot fix some extreme inconsistencies, and this is exactly where the FFT operations takes part in. Please refer to Fig.A11 for details

[歸藏(guizang.ai) @op7418]

南洋理工发布了一个可以大幅提高AI视频生成中内容一致性的方法FreeInit，演示看起来非常流畅。而且可以跟现有的SD生态结合。
他们还发了跟Animatediff结合的方法，等有大佬做插件就可以用了。视频是使用了FreeInit和未使用FreeInit的Animaetdiff的对比。

简介：
我们深入研究了视频扩散模型的噪声初始化，并发现了一个隐含的训练-推断差距，导致了推断质量的下降。
我们的关键发现是：1）推断时初始潜���量的信噪比（SNR）的时空频率分布与训练时本质上不同，2）去噪过程受到初始噪声的低频分量的显著影响。
受到这些观察的启发，我们提出了一种简洁而有效的推断采样策略FreeInit，显著改善了扩散模型生成的视频的时间一致性。
通过在推断过程中迭代地优化初始潜变量的时空低频分量，FreeInit能够弥补训练和推断之间的初始化差距，从而有效改善了生成结果的主体外观和时间一致性。

原理：
提出了FreeInit来弥合视频扩散模型训练和推断之间的初始化差距。FreeInit以迭代方式改进推断初始噪声。通过DDIM采样、DDPM前向和噪声重新初始化，初始噪声的低频成分逐渐得到改进，从而持续增强时间一致性和主体外观。

项目地址：https://t.co/Vgr7UadOF3

[Tianxing Wu @_tianxing]

@ziqi_huang_ @Gradio @scy994 @Jiang_Yuming @liuziwei7 XD

[Ziwei Liu @liuziwei7]

📢Another Free Lunch for GenAI📢

We propose #FreeInit, a sampling strategy to improve temporal consistency of video generation at inference time, requiring no training and can be plugged into any diffusion model

- Project: https://t.co/6KNCrs5C8p
- Code: https://t.co/olvF6oiYs0

[Tianxing Wu @_tianxing]

@Truongtoc1980 @_akhaliq Also, our core observation is: the quality of the initial noise's low frequency components is crucial for consistency. FreeInit is just one concise attempt, I think more approaches can be developed to address this problem.