Fu-Yun

Building generative models for images, video, and worlds.

[Fu-Yun @fywang0126]

Our solution: Joint RL Training of VLM + DiT
✅ VLM rewrites prompts → maintains quality while significantly boosting diversity ✅ VLM naturally evolves into an effective PE module ✅ For editing tasks, VLM combines source image + user instruction to generate more precise edit.

[Fu-Yun @fywang0126]

In practice, Prompt Enhancement (PE) is widely used in image/video generation, but RL-trained models fail to benefit — or even show negative gains.

[Mengdi Wang @MengdiWang10]

Rectified Diffusion: Straightness is not what your need!

We rethink rectified flow & show that matched noise-sample pairs (not straightness!) are key. No longer restricted to flow-matching—just simpler training & faster sampling for general diffusion models!

Tested on Stable Diffusion v1-5 & SDXL, our method outperforms InstaFlow at lower cost.

🔥 More speed, less compute.
📜 Read more: https://t.co/VMAbcqqFnR #AI #DiffusionModels #MachineLearning #GenerativeAI

[Ling Yang @LingYang_PU]

6 papers accepted at #ICLR2025! Highlighting two key works: IterComp (text-to-image optimization) and Rectified Diffusion (accelerating diffusion models).

Thanks to @MengdiWang10 for the guidance.

1. https://t.co/RO1rIAM9cu
2. https://t.co/xYp7bz9uyp

[Zhenjun Zhao @zhenjun_zhao]

GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields through Efficient Dense 3D Point Tracking

@wkbian, Zhaoyang Huang, Xiaoyu Shi, Yijin Li, @fywang0126, @LiHongsheng_

tl;dr: dense 3D point-tracking->pseudo 4DGS field->pretrained Video DiT

https://t.co/jowBR7sEtz

[Hao AI Lab @haoailab]

🎥 Frustrated by Sora's credit limits? Still waiting for Veo 2?
🚀 Open-source video DiTs are actually on par. We introduce FastVideo, an open-source stack to support fast video generation for SoTA open models. We have supported Mochi and Hunyuan, 8x faster inference, 720P 5-second video in 62 seconds.

[Fu-Yun @fywang0126]

@supremeZhuang Just read it, interesting observation! 😃

[The AI Timeline @TheAITimeline]

Rectified Diffusion: Straightness Is Not Your Need in Rectified Flow

Overview:
Diffusion models, while effective for visual generation, suffer from slow generation speeds.

This paper identifies that the success of rectified flow approaches, which aim to accelerate generation by straightening ODE paths, primarily depends on using pretrained models to obtain matched noise-sample pairs for retraining, making other components like flow-matching and v-prediction unnecessary.

The proposed Rectified Diffusion method focuses on achieving a curved first-order approximate ODE path suitable for models like DDPM and Sub-VP, as opposed to emphasizing straightness.

It simplifies the training process compared to previous rectified flow methods, achieving superior performance and reduced training costs, as demonstrated on benchmarks like Stable Diffusion v1-5 and Stable Diffusion XL.

Paper:
https://t.co/1hUfU5ab8J

[arXivGPT @arXivGPT]

🏷️:Rectified Diffusion: Straightness Is Not Your Need in Rectified Flow

🔗:https://t.co/hnOUlq0YRp

[Ricky T. Q. Chen @RickyTQChen]

I and Yaron @lipmanya are hiring PhD research interns for 2025 in New York City, to work on developing core foundational methods for generative modeling at scale.

If you're familiar with some of our works, shoot us an email: {rtqichen,ylipman}@meta.com

https://t.co/czVxelBDOW

[Ethan @torchcompiled]

Made a post covering Diffusion Transformers, do they deserve the hype? What makes them good? Are non-transformer methods worthless now?
Also me ranting about how awful patch embedding methods are and some experiments showing some really annoying weaknesses of them!

link below

[Fu-Yun @fywang0126]

@dreamingtulpa Thanks. For open-sourcing, it depends and is hard to say, especially considering the high training costs for videos. Some authors may prefer to remain anonymous during peer review. However, our projects AnimateLCM and PCM are open-sourced, though they are still in the preprint.

[Fu-Yun @fywang0126]

Our work Motion-I2V has been accepted by #SIGGRAPH2024! 🎉🥰

We propose a novel two-stage image-to-video generation framework, which offers more controllability of our video generation process. Our code is available at: https://t.co/DGDcq5MxaR

#SIGGRAPH2024 #MotionI2V

[Dreaming Tulpa 🥓👑 @dreamingtulpa]

Man, I pray this one gets open-sourced! 🙏

Motion-I2V can control video generation like RunwayML's Motion Brush and adds one on top by adding Motion Drag controls.

Results look incredible 🔥

[Fu-Yun @fywang0126]

@ShadyAbh @RGiryes IC. Thanks for the explanation. But it seems that it lacks the flexibility to choose arbitrary number of inference steps like diffusion? Do I understand it correctly? I will more prefer it if we can flexibly move among arbitrary timesteps and resolutions just like diffusions.

[Fu-Yun @fywang0126]

@jbhuang0604 I like these papers that they provide different perspectives to understand "how we move toward our data". Also I believe the current widely applied gaussian noise is not the best choice. More general corruption will help us deal with different data.

[Fu-Yun @fywang0126]

Our video outpainting work is accepted by #ECCV2024. 🎉. My first project on generative vision. Our method works on video outpainting across various resolutions and domains with simple tuning of the source video. Check out our code at https://t.co/Ul852tDlrF.

[AK @_akhaliq]

Be-Your-Outpainter

Mastering Video Outpainting through Input-Specific Adaptation

Video outpainting is a challenging task, aiming at generating video content outside the viewport of the input video while maintaining inter-frame and intra-frame consistency. Existing methods

[Fu-Yun @fywang0126]

Actually some previous work mentioned similar techniques but not this kind of general explanation. I share it here: https://t.co/VZMH9K1GZp.

[Fu-Yun @fywang0126]

After thoroughly reading the paper, I believe this explanation of CFG is the closest to its true nature. I have always thought CFG is a weird design, but now I finally understand why CFG works!

[Ivan Skorokhodov @isskoro]

https://t.co/YdDtO6QAIN — a  new (unusually simple and short) paper by Karras et al. with a kinda crazy idea for a much better diffusion guidance technique. It is similar to CFG, but instead of an unconditional model, they use a smaller version of the base diffusion itself (i.e., same architecture, same data, etc.).

The finding/intuition is that if a large diffusion model makes a denoising error in some point, then its small version would also make a similar mistake there, but with a larger magnitude. That's why:
- if the small and large models agree on a denoising direction, then the guidance has basically no effect
- if they disagree — then the difference between their predictions tells us the direction of the denoising error itself, and we should move backwards along it

One reason I like the paper is because it kinda proves the bizarre phenomenon (that the denoising errors of a small and large diffusions correlate) by the fact that it works. Imagine that we decompose the denoising directions for small/large diffusions as \log p_0(x) = d + e_0 and \log p_1(x) = d + e_1, where d is the GT score, and e_0 and e_1 are errors. Then, the more e_1 and e_0 correlate, the better guidance correction you get from e_1 - e_0