jo.schb ✈️CVPR

PhD student @ CompVis Group, LMU Munich

[jo.schb ✈️CVPR @jo_schb]

Diffusion models treat every part of an image equally.
→ Same number of steps. Same compute.

But images aren’t uniform. 🤔

Some regions are easy, others are hard.

So why force the model to treat them the same? 🧵

[jo.schb ✈️CVPR @jo_schb]

Probabilistic Precipitation Nowcasting with Rectified Flow Transformers

🕐Sat 6th, 4:45 - 6:45 PM
📍ExHall A & F 401

https://t.co/XTVolpsQKM

[jo.schb ✈️CVPR @jo_schb]

⚠️ Standard first stages are not sufficient for safety-critical applications!

The most extreme weather events are often the hardest to decode.

One latent → many plausible reconstructions

Deterministic decoders hide that uncertainty.

Meet FREUD 🧵👇

[jo.schb ✈️CVPR @jo_schb]

I am at @CVPR in Denver this week.

If you’re around and want to chat, feel free to send me a DM or stop by one of the two posters I’ll be presenting👇

[jo.schb ✈️CVPR @jo_schb]

Denoising, Fast and Slow: Difficulty-Aware Adaptive Sampling for Image Generation

🕐Sun 7th, 3:30 PM - 5:30 PM
📍ExHall A 658

https://t.co/AEefSVCmgS

[jo.schb ✈️CVPR @jo_schb]

Diffusion models treat every part of an image equally.
→ Same number of steps. Same compute.

But images aren’t uniform. 🤔

Some regions are easy, others are hard.

So why force the model to treat them the same? 🧵

[jo.schb ✈️CVPR @jo_schb]

Joint co-first work with @Ragor_ , and collaborators @rmsnorm, Timy Phan. Supervised by Björn Ommer @ CompVis Group LMU Munich

[jo.schb ✈️CVPR @jo_schb]

⚠️ Standard first stages are not sufficient for safety-critical applications!

The most extreme weather events are often the hardest to decode.

One latent → many plausible reconstructions

Deterministic decoders hide that uncertainty.

Meet FREUD 🧵👇

[jo.schb ✈️CVPR @jo_schb]

For more information:

Project page: https://t.co/gadRemKhzj
Paper: https://t.co/S1LPfEKaSL
Code: https://t.co/xxCQWJLJtJ

[Stefan Baumann✈️CVPR🔜SF @StefanABaumann]

The internet is full of video. So why can't novel view synthesis just scale on it?

Real-world video is simultaneously unposed, messy, and dynamic, breaking self-supervised NVS.

We fixed that. RayDer learns static-scene NVS from dynamic internet video, scaling like an LLM. A🧵

[jo.schb ✈️CVPR @jo_schb]

@JiaweiYang118 In Fig. 11 we show that we need fewer NFE to get the same or a better FID than other samplers (ODE, SDE).

[jo.schb ✈️CVPR @jo_schb]

Diffusion models treat every part of an image equally.
→ Same number of steps. Same compute.

But images aren’t uniform. 🤔

Some regions are easy, others are hard.

So why force the model to treat them the same? 🧵

[jo.schb ✈️CVPR @jo_schb]

@cloneofsimo Thanks🫶

[jo.schb ✈️CVPR @jo_schb]

@ChurchillMic @rishabh16_ You mean like per-patch tokens over both space and video time?

[jo.schb ✈️CVPR @jo_schb]

@JiaweiYang118 We mainly focus on improving generation with a given budget. But we also show empirically that we can a) achieve better performance with better allocated compute (lower NFE) and b) are orthogonal to caching

[jo.schb ✈️CVPR @jo_schb]

Joint work with @MingGui725184, @Yusong53080064 , @PingchuanMa4, @felix_m_krause, and Björn Ommer! 🫶

[jo.schb ✈️CVPR @jo_schb]

Takeaway 🚀

• Diffusion shouldn’t treat all regions equally
• Patch-wise timesteps improve performance, if done right
• Allocating compute where it matters gives further gains

Project Page: https://t.co/CNFqzK7DoG
arXiv: https://t.co/s56wDM5pSx
Code: https://t.co/uos07FxTSd