Jaesung.Choe

Research scientist @NVIDIA.

[Jon Barron @jon_barron]

I've been seeing lots of takes about the 3D Gaussian Splatting SIGGRAPH paper (https://t.co/vA8ezuluEz) that compare it to NeRF in a way that, in my opinion, undersells how cool the Gaussian Splatting paper is. Here's how I'm thinking about this stuff.

[Ziwei Liu @liuziwei7]

We propose 🔥HumanLiff🔥, a layer-wise 3D human generative model with unified diffusion process, which disentangles and controls the generation of human body and various clothes.

- Project: https://t.co/PWIkSd4NAB
- Paper: https://t.co/dUe7f7ZeTH
- Code: https://t.co/jOVNJomLfH

[Minhyuk Sung @MinhyukSung]

Our SALAD🥗 is powering the text-to-3D service at
@3dpresso_ai by ReconLabs.

Check out this webpage:
https://t.co/27mNWXYtMD

SALAD @ICCVConference
Project: https://t.co/S0YuRaSdQF
arXiv: https://t.co/Oz0d4948JH
HuggingFace: https://t.co/5LKDXt9Sx6

#SALAD3D #3Dpresso #ICCV2023

[Angela Dai @angelaqdai]

Check out @ABokhovkin's Mesh2Tex!

From a real-world image and a shape mesh to texture, we generate high-res and realistic texturing without requiring any matching geometry or pose alignment to the image!
w/ @shubtuls
#ICCV2023
https://t.co/mdaErke58q
https://t.co/U8UV5YKTKJ

[Michael Black @Michael_J_Black]

I'm excited about TADA! because it makes it easy to create beautiful avatars using only text. Unlike methods based on neural shape, TADA! avatars are fully compatible with SMPL-X, meaning that they can be animated and integrated into existing game engines. Check out the thread.

[Yuliang Xiu @yuliangxiu]

Yes, we're also developing Text2Avatar, but TADA has distinct advantages:

1. Simplicity: TADA uses SMPL-X + displacement layer, no NeRF/NeuS needed.

2. Alignment (geometry & texture): TADA ensures semantic alignment on face and pattern alignment on clothing.

[#ICCV2025 @ICCVConference]

#ICCV2023 camera-ready instructions.

Final submission (completion of all steps) is August 18, 2023, at 11:59 PM Pacific Daylight Time.

https://t.co/1L85MiSeqz

[AK @_akhaliq]

3D Gaussian Splatting for Real-Time Radiance Field Rendering

paper page: https://t.co/q9P2DmfJyP

Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (>= 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.

[Ziwei Liu @liuziwei7]

We are organizing the #OmniObject3D challenge @ICCVConference with two competition tracks:

1) Track 1: sparse-view 3D reconstruction
2) Track 2: 3D object generation

- Challenge period: Aug 1 - Sep 15, 2023
- Homepage: https://t.co/OIddYiTLZk
- CodaLab: https://t.co/s7hR2mjhQq

[Matthias Niessner @MattNiessner]

(1/2)
Happy to announce that Text2Tex has been accepted at #ICCV2023 🎉

Taking a mesh and a text prompt as input, Text2Tex generates high quality textures - it's fully automated and easy to scale to many models!

Project: https://t.co/6Hp9zGsaPH
Video: https://t.co/fu0awrihBE

[Matt Deitke @mattdeitke]

Introducing Objaverse-XL, an open dataset of over 10 million 3D objects!

With it, we train Zero123-XL, a foundation model for 3D, observing incredible 3D generalization abilities: 🧵👇

📝 Paper: https://t.co/2oNakoka7v

[AK @_akhaliq]

Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation

paper page: https://t.co/r3Dq5tXJRh

Large text-to-image diffusion models have exhibited impressive proficiency in generating high-quality images. However, when applying these models to video domain, ensuring temporal consistency across video frames remains a formidable challenge. This paper proposes a novel zero-shot text-guided video-to-video translation framework to adapt image models to videos. The framework includes two parts: key frame translation and full video translation. The first part uses an adapted diffusion model to generate key frames, with hierarchical cross-frame constraints applied to enforce coherence in shapes, textures and colors. The second part propagates the key frames to other frames with temporal-aware patch matching and frame blending. Our framework achieves global style and local texture temporal consistency at a low cost (without re-training or optimization). The adaptation is compatible with existing image diffusion techniques, allowing our framework to take advantage of them, such as customizing a specific subject with LoRA, and introducing extra spatial guidance with ControlNet. Extensive experimental results demonstrate the effectiveness of our proposed framework over existing methods in rendering high-quality and temporally-coherent videos.

[AK @_akhaliq]

Interpolating between Images with Diffusion Models

paper page: https://t.co/h3WJiNDCmt

One little-explored frontier of image generation and editing is the task of interpolating between two input images, a feature missing from all currently deployed image generation pipelines. We argue that such a feature can expand the creative applications of such models, and propose a method for zero-shot interpolation using latent diffusion models. We apply interpolation in the latent space at a sequence of decreasing noise levels, then perform denoising conditioned on interpolated text embeddings derived from textual inversion and (optionally) subject poses. For greater consistency, or to specify additional criteria, we can generate several candidates and use CLIP to select the highest quality image. We obtain convincing interpolations across diverse subject poses, image styles, and image content, and show that standard quantitative metrics such as FID are insufficient to measure the quality of an interpolation.

[AK @_akhaliq]

3D-LLM: Injecting the 3D World into Large Language Models

paper page: https://t.co/mQhF3C9y0U

Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

[Cezary Sobociński @CezaryXR]

To be honest, Meta's codec avatars are probably one of the most exciting and interesting features that will come to VR/AR headsets in the next few years.

I can't wait to see updates on Meta Connect this year

[AK @_akhaliq]

TokenFlow: Consistent Diffusion Features for Consistent Video Editing

paper page: https://t.co/xCmpqSztMf

The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.

[Paul-Edouard Sarlin @pesarlin]

You liked SuperGlue? You'll love ⚡️LightGlue⚡️, our new deep network for light-speed image matching!
➡️Faster, stronger, easier to train than SuperGlue
➡️Code: https://t.co/SLLvKOwbEo
➡️Paper: https://t.co/8b38JCoFdg
Fantastic work by @PhilippCSE for #ICCV2023, with @mapo1
1/

[Matthias Niessner @MattNiessner]

We released our ActorsHQ dataset from our #SIGGRAPH work: "HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion".

ActorsHQ provides high-fidelity multi-view captures at 12MP resolution from 160 cameras with per-frame mesh reconstructions.

https://t.co/WwoOBwKUYj

[Chris Choy, Ph.D. @realChrisChoy]

A new high-performance python library for physics simulation, geometry processing, and procedural animation!
https://t.co/GOHc4ko92a

[Chris Heinrich @cpheinrich]

An under appreciated application of NeRFs is camera stabilization. You are rendering novel views, but from a stabilized trajectory that is very close to the original

Nice paper on this by @Jimantha and team from CVPR

Website: https://t.co/CCDRB84Pbo