Jing Tan

phd student @ mmlab, cuhk; gen AI

[Jing Tan @jingtan_]

🚀Excited to introduce 𝗜𝗺𝗮𝗴𝗶𝗻𝗲𝟯𝟲𝟬, the 1st framework that lifts standard videos into 360 videos with rich and structured motion, unlocking dynamic scene experience from full 360 degrees. 🎥🌍

- Webpage (w/ VR): https://t.co/0YsLSEAAT4
- Arxiv: https://t.co/0nBUyB9AKs

[AK @_akhaliq]

Talk2Move

Reinforcement Learning for Text-Instructed Object-Level Geometric Transformation in Scenes

[Jing Tan @jingtan_]

@jie_liu1 杰哥牛哇！！

[Ye Fang @YeFang0110]

💡RelightVid: Temporal-Consistent Diffusion Model for Video Relighting.

A temporally consistent video relighting framework. By extending IC-Light with temporal layers and multi-modal conditions, it enables coherent and flexible video relighting.🚀https://t.co/gnHu4Tc1Db

[Zhibing Li @ZhibingLi_6626]

🎉 Excited to introduce IDArb! 🎉

Our method can predict plausible and 𝗰𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝘁 geometry and PBR material for 𝗮𝗻𝘆 𝗻𝘂𝗺𝗯𝗲𝗿📷 of input images under 𝘃𝗮𝗿𝘆𝗶𝗻𝗴 𝗶𝗹𝗹𝘂𝗺𝗶𝗻𝗮𝘁𝗶𝗼𝗻𝘀☀️ !

Webpage: https://t.co/GvfyvbEq25

[Dreaming Tulpa 🥓👑 @dreamingtulpa]

Imagine360 is a new method that can turn any video into 360° video 👀

[Jing Tan @jingtan_]

Thanks for sharing! 😃 Please do visit our project page for immersive VR interaction experience. Code comes in next month: https://t.co/FCY8Yxlx6n. Stay tuned!

[MrNeRF @janusch_patas]

Imagine360: Immersive 360 Video Generation from Perspective Anchor

Check out the project page, highly recommended❗️

Abstract (excerpt):
We introduce Imagine360, the first perspective-to-360∘ video generation framework that creates high-quality 360∘ videos with rich and diverse motion patterns from video anchors.

Imagine360 learns fine-grained spherical visual and motion patterns from limited 360∘ video data with several key designs.

1) Firstly we adopt the dual-branch design, including a perspective and a panorama video denoising branch to provide local and global constraints for 360∘ video generation, with motion module and spatial LoRA layers fine-tuned on extended web 360∘ videos.

2) Additionally, an antipodal mask is devised to capture long-range motion dependencies, enhancing the reversed camera motion between antipodal pixels across hemispheres.

3) To handle diverse perspective video inputs, we propose elevation-aware designs that adapt to varying video masking due to changing elevations across frames.
Extensive experiments show Imagine360 achieves superior graphics quality and motion coherence among state-of-the-art 360∘ video generation methods. We believe Imagine360 holds promise for advancing personalized, immersive 360∘ video creation.

[Jing Tan @jingtan_]

We plan to release the code and weights next month. Stay tuned!
Code repo 👉 https://t.co/FCY8Yxlx6n

[Jing Tan @jingtan_]

🚀Excited to introduce 𝗜𝗺𝗮𝗴𝗶𝗻𝗲𝟯𝟲𝟬, the 1st framework that lifts standard videos into 360 videos with rich and structured motion, unlocking dynamic scene experience from full 360 degrees. 🎥🌍

- Webpage (w/ VR): https://t.co/0YsLSEAAT4
- Arxiv: https://t.co/0nBUyB9AKs

[Jing Tan @jingtan_]

Excited to see this incredible progress in 3D panoramic scene generation! 🚀 We also have a preliminary attempt in this direction. Take a look and share your thoughts! 🙌
- LayerPano3D Paper: https://t.co/0er9iV1Na1
- Project page: https://t.co/p8PmzVZMyf

[World Labs @theworldlabs]

We’ve been busy building an AI system to generate 3D worlds from a single image. Check out some early results on our site, where you can interact with our scenes directly in the browser!

https://t.co/ASD6ZHMwxI

1/n

[Ziwei Liu @liuziwei7]

🔥Text to Explorable 3D Panorama🔥

#LayerPano3D is a framework for full-view, explorable panoramic 🏝️3D scene generation🏝️ from a single text prompt.

- Project: https://t.co/XhTHHyr2D8
- Code: https://t.co/uKoTOyBk1n
- Video: https://t.co/8tZpOVBw2r

@Gradio demo coming soon :)

[Shuai Yang @yangshuai1227]

Thanks a lot for sharing! We will soon open-source our code at https://t.co/YX1xn9PqY8. Stay tuned!

[Jing Tan @jingtan_]

Thanks for sharing our work! LayerPano3D generates 360x180 full view, explorable panoramic 3D scene from a single text prompt. More examples are available in our project page: https://t.co/p8PmzVZMyf

[AK @_akhaliq]

LayerPano3D

Layered 3D Panorama for Hyper-Immersive Scene Generation

discuss: https://t.co/ibqSFaB74o

3D immersive scene generation is a challenging yet critical task in computer vision and graphics. A desired virtual 3D scene should 1) exhibit omnidirectional view consistency, and 2) allow for free exploration in complex scene hierarchies. Existing methods either rely on successive scene expansion via inpainting or employ panorama representation to represent large FOV scene environments. However, the generated scene suffers from semantic drift during expansion and is unable to handle occlusion among scene hierarchies. To tackle these challenges, we introduce LayerPano3D, a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt. Our key insight is to decompose a reference 2D panorama into multiple layers at different depth levels, where each layer reveals the unseen space from the reference views via diffusion prior. LayerPano3D comprises multiple dedicated designs: 1) we introduce a novel text-guided anchor view synthesis pipeline for high-quality, consistent panorama generation. 2) We pioneer the Layered 3D Panorama as underlying representation to manage complex scene hierarchies and lift it into 3D Gaussians to splat detailed 360-degree omnidirectional scenes with unconstrained viewing paths. Extensive experiments demonstrate that our framework generates state-of-the-art 3D panoramic scene in both full view consistency and immersive exploratory experience. We believe that LayerPano3D holds promise for advancing 3D panoramic scene creation with numerous applications.

[AK @_akhaliq]

LayerPano3D

Layered 3D Panorama for Hyper-Immersive Scene Generation

discuss: https://t.co/ibqSFaB74o

3D immersive scene generation is a challenging yet critical task in computer vision and graphics. A desired virtual 3D scene should 1) exhibit omnidirectional view consistency, and 2) allow for free exploration in complex scene hierarchies. Existing methods either rely on successive scene expansion via inpainting or employ panorama representation to represent large FOV scene environments. However, the generated scene suffers from semantic drift during expansion and is unable to handle occlusion among scene hierarchies. To tackle these challenges, we introduce LayerPano3D, a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt. Our key insight is to decompose a reference 2D panorama into multiple layers at different depth levels, where each layer reveals the unseen space from the reference views via diffusion prior. LayerPano3D comprises multiple dedicated designs: 1) we introduce a novel text-guided anchor view synthesis pipeline for high-quality, consistent panorama generation. 2) We pioneer the Layered 3D Panorama as underlying representation to manage complex scene hierarchies and lift it into 3D Gaussians to splat detailed 360-degree omnidirectional scenes with unconstrained viewing paths. Extensive experiments demonstrate that our framework generates state-of-the-art 3D panoramic scene in both full view consistency and immersive exploratory experience. We believe that LayerPano3D holds promise for advancing 3D panoramic scene creation with numerous applications.

[Jing Tan @jingtan_]

We are excited to introduce LayerPano3D, a novel framework to generate full-view, explorable panoramic 3D scene from a single text prompt! More examples are in our project page.
✨Project: https://t.co/p8PmzVZeIH
✨Paper: https://t.co/0er9iV1fkt
✨Code: https://t.co/LEFIiUE69i

[Zhenzhi Wang @zhenzhiwang]

Want to generate camera controllable human videos like a real movie clip? Try our HumanVid dataset and a baseline model combined by AnimateAnyone and CameraCtrl.
Project Page: https://t.co/ix2w7jYelN
Paper: https://t.co/D8uCejx6KZ
Data and code coming soon.

[Simon Lee @simonxxoo]

WIP

#b3d

[Tanishq Mathew Abraham, Ph.D. @iScienceLuvr]

I will attempt to explain the basic idea of how diffusion models work!

... in only 15 tweets! 😲

Let's get started ↓

[Christian Wolf (🦋🦋🦋) @chriswolfvision]

If you are desperate for space in your #NeurIPS2022 paper, eg because reviewers asked to include parts of your rebuttal into the main paper, one possibility is to enclose some $$ ... $$ equations with \textstyle{ ... }. This will typeset them as if they were inline $ ... $:

[Tzu-Mao Li @tzumaoli]

Read older papers (before 2000). Many important insights already appeared decades ago. Don't just read from certain groups. This tends to miss the big picture. Read from broader fields. Read statistics, robotics, physics, applied math, graphics, PL, databases, linguistics, HCI.

[Jia-Bin Huang @jbhuang0604]

Just realized that I now have 40 (!) twitter threads! 🤩

I organized them into "Awesome Tips": https://t.co/5CTTuJm3Jg

Really appreciate for all your comments, critics, likes, and retweets. Hope these threads are useful!

What should I write next? 🤔