#VideoGrain - Twitter Hashtag

over 1 year ago

#VideoGrain #ICLR2025 #VideoGeneration #artificalintelligence 📷 Details (3/n) - Method So our key insights and designs are: (1) Text-to-region control. As seen in the figure mid, we modulate cross-attn and redistribute each local prompt’s weight to align with target areas, enabling precise text-to-region control. (2) Keep feature separation. As seen in the figure bottom, we modulate self-attn, the query feature from the left man’s nose (e.g., p) attends only to the left instance, avoiding distraction to the right instance. We break the diffusion model’s class-level feature correspondence, ensuring feature separation at the instance level.

Ayden_Yang_'s tweet photo. #VideoGrain #ICLR2025 #VideoGeneration #artificalintelligence

📷 Details (3/n) - Method

So our key insights and designs are:

(1) Text-to-region control. As seen in the figure mid, we modulate cross-attn and redistribute each local prompt’s weight to align with target areas, enabling precise text-to-region control.

(2) Keep feature separation. As seen in the figure bottom, we modulate self-attn, the query feature from the left man’s nose (e.g., p) attends only to the left instance, avoiding distraction to the right instance.
We break the diffusion model’s class-level feature correspondence, ensuring feature separation at the instance level.

xiangpeng yang @Ayden_Yang_

over 1 year ago

🚀 #ICLR2025 Thanks @_akhaliq @Gradio share. We just open-sourced VideoGrain, a training-free, multi-grained (class/instance/part) video editing method Check it out: 📄ArXiv: https://t.co/q700znUrCh 🎥Project: https://t.co/5vIvJmVn7Y 💻Code: https://t.co/bWfbH6hrjs

0

12

4

7

2K

0

2

0

132

xiangpeng yang @Ayden_Yang_

over 1 year ago

#VideoGrain #ICLR2025 #VideoGeneration #ArtificialIntelligence 📷 Details (2/n) - Motivation So why diffusion model fail at multi-grained video editing? 1) As seen in (b), the clustering inversion self-attn feature shows that although it captures a clear semantic layout, it fails to distinguish between distinct instances. (e.g., “left man” and “right man”). 2) Direct editing same-class instances fails since the cross-attn weight distribution is not right, as shown in (d). i.e., "Iron Man" and "Spiderman" weights overlap on the left, and “Blossom” weight leaks onto the right man, resulting in the failed edit in (c).

Ayden_Yang_'s tweet photo. #VideoGrain #ICLR2025 #VideoGeneration #ArtificialIntelligence
📷 Details (2/n) - Motivation

So why diffusion model fail at multi-grained video editing?

1) As seen in (b), the clustering inversion self-attn feature shows that although it captures a clear semantic layout, it fails to distinguish between distinct instances. (e.g., “left man” and “right man”).

2) Direct editing same-class instances fails since the cross-attn weight distribution is not right, as shown in (d). i.e., "Iron Man" and "Spiderman" weights overlap on the left, and “Blossom” weight leaks onto the right man, resulting in the failed edit in (c).

xiangpeng yang @Ayden_Yang_

over 1 year ago

🚀 #ICLR2025 Thanks @_akhaliq @Gradio share. We just open-sourced VideoGrain, a training-free, multi-grained (class/instance/part) video editing method Check it out: 📄ArXiv: https://t.co/q700znUrCh 🎥Project: https://t.co/5vIvJmVn7Y 💻Code: https://t.co/bWfbH6hrjs

0

12

4

7

2K

0

4

0

131

xiangpeng yang @Ayden_Yang_

over 1 year ago

#VideoGrain 📷 Details (1/n)👇 What is multi-grained video editing? As seen in the fig left: (1) Class-level: Editing objects within the same class. (2) Instance-level: Editing each individual instance to distinct object. (3) Part-level: Adding new objects or modifying existing attributes at the part-level However, as seen in the fig right, previous SOTA failed at the instance level, like Pika.

Ayden_Yang_'s tweet photo. #VideoGrain
📷 Details (1/n)👇

What is multi-grained video editing?
As seen in the fig left:

(1) Class-level: Editing objects within the same class.

(2) Instance-level: Editing each individual instance to distinct object.

(3) Part-level: Adding new objects or modifying existing attributes at the part-level

However, as seen in the fig right, previous SOTA failed at the instance level, like Pika.

xiangpeng yang @Ayden_Yang_

over 1 year ago

🚀 #ICLR2025 Thanks @_akhaliq @Gradio share. We just open-sourced VideoGrain, a training-free, multi-grained (class/instance/part) video editing method Check it out: 📄ArXiv: https://t.co/q700znUrCh 🎥Project: https://t.co/5vIvJmVn7Y 💻Code: https://t.co/bWfbH6hrjs

0

12

4

7

2K

0

2

0

56

whitey @ohmanWhitey

over 3 years ago

@talkindawson Ep310 cont'd #CotW #FloodOfEmotions #IdiotTrainwrecks #VideoGrain #CanadianTreasure @jannarden #MushroomHair #ChowderMan #8lbGoblin #Pissy #AJSucks #EmailBitch #AJLovesNSYNC

1

3

1

0

114

Top Tweets for #VideoGrain

Last Seen Hashtags on Sotwe

Trends for you

Most Popular Users