OpenAI released a technical report on Sora:
- method for turning visual data of all types into a unified representation that enables large-scale training of generative models
- qualitative evaluation of Sora’s capabilities and limitations.
https://t.co/m8VJ6KPBlr
Given a reference RGB image and depth map, Stylized Radiance Fields with Geometry Transfer (Geo-SRF) can stylize the geometry of a radiance field.
It also supports interpolation between the original scene and stylized output.
🔗 https://t.co/vl0GDU9Iho
We introduce a new method to stylize radiance fields using geometry transfer. We propose to use depth maps as style guides and utilize deformation fields to ensure coherent stylization of both shape and appearance.
Website: https://t.co/uyF4a1MYVB
Generate Anything Anywhere in Any Scene
paper page: https://t.co/rxA46oMuh8
Text-to-image diffusion models have attracted considerable interest due to their wide applicability across diverse fields. However, challenges persist in creating controllable models for personalized object generation. In this paper, we first identify the entanglement issues in existing personalized generative models, and then propose a straightforward and efficient data augmentation training strategy that guides the diffusion model to focus solely on object identity. By inserting the plug-and-play adapter layers from a pre-trained controllable diffusion model, our model obtains the ability to control the location and size of each generated personalized object. During inference, we propose a regionally-guided sampling technique to maintain the quality and fidelity of the generated images. Our method achieves comparable or superior fidelity for personalized objects, yielding a robust, versatile, and controllable text-to-image diffusion model that is capable of generating realistic and personalized images. Our approach demonstrates significant potential for various applications, such as those in art, entertainment, and advertising design.
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization
paper page: https://t.co/agAFFqC72T
Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.
RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models
Xingchen Zhou, Ying He, F. Richard Yu, Jianqiang Li, You Li
tl;dr: NeRF+diffusion model->edit 3D content in NeRF
https://t.co/cloCvgDJuM
MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion
@Gradio demo: https://t.co/pXerXl7LZf
This paper introduces MVDiffusion, a simple yet effective multi-view image generation method for scenarios where pixel-to-pixel correspondences are available, such as perspective crops from panorama or multi-view images given geometry (depth maps and poses). Unlike prior models that rely on iterative image warping and inpainting, MVDiffusion concurrently generates all images with a global awareness, encompassing high resolution and rich content, effectively addressing the error accumulation prevalent in preceding models. MVDiffusion specifically incorporates a correspondence-aware attention mechanism, enabling effective cross-view interaction. This mechanism underpins three pivotal modules: 1) a generation module that produces low-resolution images while maintaining global correspondence, 2) an interpolation module that densifies spatial coverage between images, and 3) a super-resolution module that upscales into high-resolution outputs. In terms of panoramic imagery, MVDiffusion can generate high-resolution photorealistic images up to 1024x1024 pixels. For geometry-conditioned multi-view image generation, MVDiffusion demonstrates the first method capable of generating a textured map of a scene mesh.
GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields
Barbara Roessle, Norman Müller, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, @MattNiessner
tl;dr: GAN+NeRF->multi-view patch-based rendering constraints->NeRF
https://t.co/xIyYpiRbmE
Introducing “FlowCam: Training Generalizable 3D Radiance Fields w/o Camera Poses via Pixel-Aligned Scene Flow”! We train a generalizable 3D scene representation self-supervised on datasets of raw videos, without any pre-computed camera poses or SFM!
https://t.co/y4dS5z2JpR
1/n
Adjustable Visual Appearance for Generalizable Novel View Synthesis
Josef Bengtson, David Nilsson, Che-Tsung Lin, Marcel Büsching, Fredrik Kahl
tl;dr: transformer based generalizable novel view synthesis method that allows for visual appearance changes
https://t.co/kdMwoWew5y
Have to recommend Jia-Bin Huang's tweetorial on making an overview figure. This is a thing getting popularized in AI/ML papers (fastest paced in all literature) but apparently a good practice everywhere, even for grant proposals. https://t.co/4Fqzd8nPgF
The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation
paper page: https://t.co/2KG9qBiFOS
Denoising diffusion probabilistic models have transformed image generation with their impressive fidelity and diversity. We show that they also excel in estimating optical flow and monocular depth, surprisingly, without task-specific architectures and loss functions that are predominant for these tasks. Compared to the point estimates of conventional regression-based methods, diffusion models also enable Monte Carlo inference, e.g., capturing uncertainty and ambiguity in flow and depth. With self-supervised pre-training, the combined use of synthetic and real data for supervised training, and technical innovations (infilling and step-unrolled denoising diffusion training) to handle noisy-incomplete training data, and a simple form of coarse-to-fine refinement, one can train state-of-the-art diffusion models for depth and optical flow estimation. Extensive experiments focus on quantitative performance against benchmarks, ablations, and the model's ability to capture uncertainty and multimodality, and impute missing values. Our model, DDVM (Denoising Diffusion Vision Model), obtains a state-of-the-art relative depth error of 0.074 on the indoor NYU benchmark and an Fl-all outlier rate of 3.26\% on the KITTI optical flow benchmark, about 25\% better than the best published method.
DäRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation
Jiuhn Song, Seonghoon Park, Honggyu An, Seokju Cho, Min-Seop Kwak, Sungjin Cho, Seungryong Kim
tl;dr: MDE-enhanced NeRF at both seen and unseen views
https://t.co/xdcURfYocj
Control4D: Dynamic Portrait Editing by Learning 4D GAN from 2D Diffusion-based Editor
propose Control4D, a novel approach for high-fidelity and temporally consistent 4D portrait editing. Control4D is built upon an efficient 4D representation with a 2D diffusion-based editor. Instead of using direct supervisions from the editor, our method learns a 4D GAN from it and avoids the inconsistent supervision signals. Specifically, we employ a discriminator to learn the generation distribution based on the edited images and then update the generator with the discrimination signals. For more stable training, multi-level information is extracted from the edited images and used to facilitate the learning of the generator. Experimental results show that Control4D surpasses previous approaches and achieves more photo-realistic and consistent 4D editing performances.
paper page: https://t.co/WVC7RThlRp
If you like the visualisations of the #ACErelocalizer, they are fully integrated in our public code release:
https://t.co/YRxzCX7H3Q
Run train and test with "--render_visualisations True" and get a nice video of the full process. Also for your data!
A few more that I like 🧵👇
Very happy to share our SIGGRAPH 2023 work "NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images".
Paper: https://t.co/JNCDfy1eCP
Project page: https://t.co/zM1mSdyJ4Z
Codes: https://t.co/LxaS1mvwR4