Bichen Wu

Founder | Ex-Meta, Google, Berkeley PhD, Tsinghua | Marathon Runner & Mountaineer · 4/7 Seven Summits

[Bichen Wu @wu_bichen]

@jiahuiyu @finkd @alexandr_wang Congrats and good luck!

[Devi Parikh @deviparikh]

End of an era.

Curious: How many of you know of Amazon Mechanical Turk?

[Bichen Wu @wu_bichen]

@pmddomingos I would interpret it differently. The market sees this as an indicator that GDM is struggling. To put it differently, if they didn’t leave and two months passed, GDM fails to deliver a good model, the same thing would happen.

[Bichen Wu @wu_bichen]

Some notes from CVPR after a few days of conversations, especially around media generation.

A lot of attention has shifted toward embodied AI, but image / video / 3D generation is still very active. There were many papers, demos, and hallway conversations around generation, but it remains difficult to tell which ideas really work.

Compute still seems to be the bottleneck, even inside labs where people assume compute is abundant. Heard that GDM had to pause resources for almost everything except coding, leaving other efforts hanging.

Architecture-wise, one trend is the move toward omni models: one model that can input and output many modalities, including image, video, audio, and text. The key question is how to enable reasoning across modalities, and how to combine agentic abilities with such models.

For media generation models, evaluation remains one of the hardest problems, since it is intrinsically subjective. Big labs still rely heavily on human annotators, and the “taste” of the model is still shaped by a small group of model builders.

Also heard about some interesting approaches around distilling personal taste into VLMs, using agentic self-loops to align VLM outputs with personal preferences. I remain skeptical about scalability and generalization, but it seems directionally interesting.

Productionization of video generation seems to have cooled down in the US. After Sora’s shutdown, not many US companies seem to still be focused on frontier video generation models. GDM looks like the main exception. Meanwhile, Chinese models such as Seedance and Kling seem to be carrying much of the effort.

Many video content creation startups are now refocusing on agent harnesses: planning and tool systems for storyboards, shot planning, character references, and consistent footage using external models like Veo or SeedDance. The concern is whether this harness has a deep enough moat.

For media GenAI, the biggest open question is: what is the next “coding”? Code generation started as a small LLM application, but quickly expanded from tab completion to software engineering to doing all knowledge works. What is the analogous non-obvious but generalizable task for image / video / media generation?

A lot of video generation models are now being rebranded or repurposed as world models, especially for generating data for embodied AI training or adapting them into action models directly. A friend told me they have seen early promising results in autonomous driving, where the control signal is relatively low-dimensional.

Still spending another day at CVPR. Will share more notes as I keep learning.

[Bichen Wu @wu_bichen]

I’ll be at CVPR in Denver 6/4–6/6. My 10th CVPR in a row. I’m interested in AI for visual storytelling, evaluation beyond verifiable problems: taste, intent, narrative quality of AI creation. Please send paper/workshop recs, and most importantly people I should catch up with.

[Bichen Wu @wu_bichen]

We are extremely excited that team Centropy will be joining forces with the legendary @ericschmidt to embark on a new adventure!

[Bichen Wu @wu_bichen]

@chenkun198282 太期待了！

[Ji Hou @j1h0u]

Registration of GenAI Media Generation Challenge #CVPR2024 is open: https://t.co/bLc65Leh1u, we provide different tracks of challenge, such as t2i/i2i etc, and the winner prize is generous.

[Bichen Wu @wu_bichen]

@lepadphone 好厉害！

[Bichen Wu @wu_bichen]

@sainingxie Great work as always, Saining!

[Bichen Wu @wu_bichen]

@op7418 这个是salesforce 做的

[Bichen Wu @wu_bichen]

Thanks @_akhaliq for featuring this work led by our research intern @LiangJeff95, where we proposed a new way of utilizing optical flows for video-to-video generation.

[AK @_akhaliq]

Meta just announced FlowVid

Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis

paper page: https://t.co/GNUdDAt1Qa

Diffusion models have transformed the image-to-image (I2I) synthesis and are now permeating into videos. However, the advancement of video-to-video (V2V) synthesis has been hampered by the challenge of maintaining temporal consistency across video frames. This paper proposes a consistent V2V synthesis framework by jointly leveraging spatial conditions and temporal optical flow clues within the source video. Contrary to prior methods that strictly adhere to optical flow, our approach harnesses its benefits while handling the imperfection in flow estimation. We encode the optical flow via warping from the first frame and serve it as a supplementary reference in the diffusion model. This enables our model for video synthesis by editing the first frame with any prevalent I2I models and then propagating edits to successive frames. Our V2V model, FlowVid, demonstrates remarkable properties: (1) Flexibility: FlowVid works seamlessly with existing I2I models, facilitating various modifications, including stylization, object swaps, and local edits. (2) Efficiency: Generation of a 4-second video with 30 FPS and 512x512 resolution takes only 1.5 minutes, which is 3.1x, 7.2x, and 10.5x faster than CoDeF, Rerender, and TokenFlow, respectively. (3) High-quality: In user studies, our FlowVid is preferred 45.7% of the time, outperforming CoDeF (3.5%), Rerender (10.2%), and TokenFlow (40.4%).

[Justine Moore @venturetwins]

2023 was a breakout year for AI video.

In January, there were no public text-to-video models. Now, there are dozens of video gen products and millions of users.

A recap of the biggest developments + companies to watch 👇

[Jonas Schult @JonasSchultCV]

🚀 Check out our new work ControlRoom3D.

ControlRoom3D creates diverse and plausible 3D room meshes aligning well with user-defined room layouts and textual descriptions of the room style.

🌐 Project Page: https://t.co/vDpGe3AED0
📄 ArXiv: https://t.co/h7aVZPOtjl

🧵 (1/8)

[Bichen Wu @wu_bichen]

Thanks @_akhaliq for featuring this. We are so excited that we are finally able to share this work with the community.

[AK @_akhaliq]

Meta just announced Fairy

Fast Parallelized Instruction-Guided Video-to-Video Synthesis

paper page: https://t.co/oTYcdJDlpt

introduce Fairy, a minimalist yet robust adaptation of image-editing diffusion models, enhancing them for video editing applications. Our approach centers on the concept of anchor-based cross-frame attention, a mechanism that implicitly propagates diffusion features across frames, ensuring superior temporal coherence and high-fidelity synthesis. Fairy not only addresses limitations of previous models, including memory and processing speed. It also improves temporal consistency through a unique data augmentation strategy. This strategy renders the model equivariant to affine transformations in both source and target images. Remarkably efficient, Fairy generates 120-frame 512x384 videos (4-second duration at 30 FPS) in just 14 seconds, outpacing prior works by at least 44x. A comprehensive user study, involving 1000 generated samples, confirms that our approach delivers superior quality, decisively outperforming established methods.

[Bichen Wu @wu_bichen]

Thanks @_akhaliq for featuring this work led by our research intern @HaoyumaU, demonstrating the power and efficiency of transformer based architectures for video editing, frame interpolation, etc. Check out the project page for more examples: https://t.co/D4oO2BaVG3.

[AK @_akhaliq]

MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers

paper page: https://t.co/JECZdMqwyc

Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in practical applications. This study addresses this challenge by breaking down the text-based video editing process into two separate stages. In the first stage, we leverage an existing text-to-image diffusion model to simultaneously edit a few keyframes without additional fine-tuning. In the second stage, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the keyframes, benefiting from structural guidance provided by intermediate frames. Our comprehensive set of experiments illustrates the efficacy and efficiency of MaskINT when compared to other diffusion-based methodologies. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.

[Bichen Wu @wu_bichen]

@petergyang Not a short term solution but for me, after losing about 20kg of weight and stared regular exercises, the back pain goes away naturally.

[Bichen Wu @wu_bichen]

Thanks @_akhaliq for sharing our work!

[AK @_akhaliq]

MixRT: Mixed Neural Representations For Real-Time NeRF Rendering

paper page: https://t.co/QZsXrz4siP

Neural Radiance Field (NeRF) has emerged as a leading technique for novel view synthesis, owing to its impressive photorealistic reconstruction and rendering capability. Nevertheless, achieving real-time NeRF rendering in large-scale scenes has presented challenges, often leading to the adoption of either intricate baked mesh representations with a substantial number of triangles or resource-intensive ray marching in baked representations. We challenge these conventions, observing that high-quality geometry, represented by meshes with substantial triangles, is not necessary for achieving photorealistic rendering quality. Consequently, we propose MixRT, a novel NeRF representation that includes a low-quality mesh, a view-dependent displacement map, and a compressed NeRF model. This design effectively harnesses the capabilities of existing graphics hardware, thus enabling real-time NeRF rendering on edge devices. Leveraging a highly-optimized WebGL-based rendering framework, our proposed MixRT attains real-time rendering speeds on edge devices (over 30 FPS at a resolution of 1280 x 720 on a MacBook M1 Pro laptop), better rendering quality (0.2 PSNR higher in indoor scenes of the Unbounded-360 datasets), and a smaller storage size (less than 80% compared to state-of-the-art methods).

[Bichen Wu @wu_bichen]

This work is accepted by @3DVconf. To try out the demo yourself and find out the technical details, check out the project page: https://t.co/mQpekYVX2w

[Bichen Wu @wu_bichen]

The coolest feature of MixRT is that it can simultaneously render images while computing collision detection, in real time, on mobile. The video attached is a screen record on my mobile phone running MixRT in a browser.