3D AI Lab

3D AI Lab @ TU Munich
Reconstruction and semantic understanding of real-world environments

[Matthias Niessner @MattNiessner]

📢TriFlow: Generating Artist-Like 3D Mesh Topology via Nearest-Vertex Vector Fields (ECCV'26)📢

Compact 3D meshes with clean, artist-like triangle topology - structured connectivity you'd expect a human modeler to make🙂

🌐https://t.co/PsrCkwUrCJ
▶️https://t.co/nlLTliWEEa

[Angela Dai @angelaqdai]

📢WorldMesh is accepted to #ECCV2026, and we're releasing the code today! 🎉

Led by @mschneider456: navigable, multi-room 3D scenes from a text prompt, with a mesh scaffold conditioning image diffusion for global consistency + photorealistic detail

👇
https://t.co/8fXCl2flIu

[Matthias Niessner @MattNiessner]

📢 GenRecon Code Release 📢

Few images in → complete, high-fidelity 3D scene out!

GenRecon builds a generative prior on full scenes, resulting in unprecedented 3D reconstruction quality.

🔗 https://t.co/1jvzgC7aBO
🌐 https://t.co/sbz1Nb0ptN
📄 https://t.co/VqL3flGBGj

[Angela Dai @angelaqdai]

📢UnfoldArt recovers articulated 3D objects from image or text!

@ElBoudjogh24002 uses 🤖multi-agent reasoning for articulation +🎥 video priors for high-fidelity geometry & interiors
→ interactable URDFs for furniture, helicopters, humanoids, & more!

👉https://t.co/qooAnccDXN

[Matthias Niessner @MattNiessner]

High-quality 3D reconstruction is still hard. In particular, with 3DGS, getting full coverage without occlusions is often impractical: training views can look great, but you can barely move around.

With Echo-2, we take a different path: reconstruction as conditional generation.

By leveraging a strong generative prior, Echo-2 creates high-fidelity digital twins from sparse input views—constrained by what was captured, while completing appearance and geometry in a coherent 3D world.

These are a few sample scenes, but the same approach scales to large environments from only a handful of images. API update incoming!

[Angela Dai @angelaqdai]

Check out our #ECCV2026 work on visibility-guided flow matching for scan completion, mesh-guided 3d world generation, artist-like 3d mesh generation, & agentic 3D world synthesis!

Congrats @QTDSMQ, @mschneider456, @hcxrli, @ErkocZiya for amazing work :)

[Snehal Jauhri @SnehalJauhri]

Happy to share that I have started a Post-Doc at the 3D AI Lab at Technical University of Munich @TU_Muenchen, working with Prof. Angela Dai @angelaqdai!

Excited to push further on 3D generation, reinforcement learning, and embodied AI!

@MunichCenterML #3DAI #EmbodiedAI

[Angela Dai @angelaqdai]

@davnov134 from @SpAItial_AI on generating physically grounded world models at the ScanNet++ workshop!

[Angela Dai @angelaqdai]

Starting with a super cool talk by @taiyasaki on explicit representations for nvs and generative synthesis!

[Angela Dai @angelaqdai]

Come by our ScanNet++ workshop @CVPR June 3 in 710, 1:00pm onwards!

5 exciting keynotes on world models, NVS, 3D gen, understanding and more, from @taiyasaki @davnov134 @orlitany @PeterHedman3 @RamananDeva and talks from NVS+semantics benchmark winners:
https://t.co/DtU3Qg3jqd

[Snehal Jauhri @SnehalJauhri]

Thrilled to share that I've defended my PhD at TU Darmstadt, summa cum laude! 🎉

Grateful to my advisor @GeorgiaChal, the PEARL Lab, and my thesis committee: Anna Rohrbach, @danfei_xu, and @Jan_R_Peters.

More energized than ever for further impact in AI, Robotics and 3D Vision!

[Matthias Niessner @MattNiessner]

📢📢GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction📢📢

Reconstructing high-fidelity 3D scenes from sparse RGB input is hard. It needs a strong 3D prior!

We reformulate multi-view scene reconstruction as conditional 3D generation over overlapping spatial chunks, lifting posed image features into a generative shape prior via 3D conditioning. As an example prior, we build on Trellis2, and train it such that its reconstruction is pixel aligned and matches from all views.

GenRecon achieves unprecedented reconstruction quality from any sparse RGB input sequence, even from a phone capture. The reconstruction also includes PBR materials which facilitates relighting and virtual object insertion.

https://t.co/1RMD40WRpz
https://t.co/u4IEi5PTtn

Amazing work by @katha_schmid, @nicolasvluetzow, Jozef, @angelaqdai

[Matthias Niessner @MattNiessner]

Always a pleasure to be back at Stanford! 🌲

It's been almost a decade since my four years here as a Visiting Professor. I was fortunate to work with so many world-class researchers who helped shape my career. Many of those same colleagues are now leading frontier labs all over the world.

Research-wise, it was a wild time. Deep learning was just starting to take over traditional computer vision, and generative methods were barely working (remember the early VAEs and GANs?). We were one of the groups pushing early 3D deep learning. It was exciting, but chaotic. There was no PyTorch, and even simple operators had to be hand-written in CUDA. Lots of fun!

Our main focus was 3D scene reconstruction and semantic scene understanding. For scanning, we used Microsoft Kinects with methods like Voxel Hashing or BundleFusion, which led to scene understanding works like ScanNet and Matterport3D. And, of course, 3D face reconstruction—which led to the legendary Face2Face paper by @JustusThies, got me on Jimmy Kimmel Live, and ultimately sparked the foundation of @synthesiaIO.

Coming back brings up so much nostalgia for those days before massive transformers took over. But beyond the research, a lot has evolved. On campus, new dorms have transformed the landscape, there's a shiny new data science building, and an incredible lineup of new CS faculty.

Palo Alto has changed, too. There’s a new bikeway on El Camino, and Cal Ave is now a pedestrian zone with many shops having changed. (Un)surprisingly, the Nuthouse didn't survive. The building sits empty like one of its many peanut shells. Still, it's a vibrant area with fantastic food, great to hang out after an intense day of research.

Die Luft der Freiheit weht!

[SpAItial AI @SpAItial_AI]

We wired 3D Gaussian Splatting into the 1999 Quake 3 engine - it's fully playable!

Echo-2 generates the game levels, and it's rendering spaces directly inside the open source version of id Tech 3.

Anyone wants to try it?

[SpAItial AI @SpAItial_AI]

Pretty wild 3D scene generations - all made from a single image 🔥🔥🔥

1) an aircraft carrying aircrafts that carry aircrafts

[SpAItial AI @SpAItial_AI]

We❤️Gaming 💕

AI-generated first-person shooter made easy: input an image -> Echo-2 generates world -> let's play!

We're already having fun playing but it'll be open soon :)

[Lei Li @craigleili]

Human-object interaction isn't just global proximity. It's a coordinated engagement between body parts and functional object parts.

Check out HOI-PAGE for realistic generation via this explicit part-level reasoning!

[Angela Dai @angelaqdai]

Excited to share HOI-PAGE, to appear at #ICML2026! 🚀

@craigleili generates 4D human-object interactions zero-shot from text

A part-affordance graph grounds interactions via LLM+video priors, enabling complex multi-person, multi-object interactions

👉https://t.co/o0UQrhMgmt

[Matthias Niessner @MattNiessner]

Large foundation models have made enormous progress in modeling language, images, and video. These systems can generate highly realistic outputs and capture complex statistical structure in data. However, they still operate on projections of the world, text sequences and 2D pixel grids, rather than the world itself.

The real world is not a sequence of text tokens or frames; the real world is inherently anchored in 3D metric space, and dynamics across time. Objects occupy space and persist over time. They interact according to physical laws. Any model that aims to support real-world intelligence, e.g., for robotics, simulation, design, or spatial computing, must capture this structure.

This is where current approaches fall short. While most video models can generate visually plausible frames, they often lack a consistent notion of the underlying scene due to limited context windows. As a result, geometry drifts, scale is ambiguous, objects appear and disappear, and interactions are not physically grounded. The model produces superficial appearance without a persistent world representation.

For many downstream applications, this is not enough.

The first step toward addressing this is modeling 3D space and keeping it consistent. A model should recover a coherent spatial representation of the scene, including layout, geometry, and scale. This not only allows the environment to be rendered from new viewpoints but also, more critically, reasoned about in metric space. If a model cannot produce a stable 3D representation, it is not grounded in the physical world, and it will fail to model the world due to its inefficient contextual memory.

However, 3D is only the beginning.

A truly useful world model must also be temporally and physically consistent. It should not only reconstruct a scene, but also simulate it, predicting how it evolves, how objects interact, and what happens under intervention. Eventually this requires moving beyond static representations toward models that capture dynamics and causality.

I believe that generative approaches are highly compelling in this context, as they can be trained on large-scale data in a self-supervised fashion. In particular, comprehensive 3D world modeling is a highly-promising path forward, since richer environmental representations directly enable deeper and more effective learning of physical reality. Crucially, such generation enforces consistency: for instance, to generate a scene across viewpoints, a model must implicitly recover its underlying 3D structure. To generate it over time, it must capture its dynamics. This forces the model to internalize the latent state of the world, including geometry, scale, materials, motion, and physical behavior.

This also highlights a limitation of purely abstract representations. High-level embeddings or action-centric models can be effective for specific tasks, but without the ability to model and simulate the world, they will eventually remain incomplete. They compress observations, but do not fully model the underlying process that generates them.

The next generation of AI systems should therefore move beyond text and pixels, and toward physically-grounded world models: models that represent space, maintain consistency over time, and enable simulation and interaction.

This is the missing layer between the physical and digital world, which will ultimately enable AI systems not just to observe the world, but to understand and operate within it.

[Angela Dai @angelaqdai]

📢Diff3r: fast feed-forward 3DGS + per-scene optimization

@liuyuehcheng predicts optimization-ready 3DGS init end to end, computing implicit gradients via Implicit Function Theorem + Gauss-Newton approximation for fast & stable results

Check it out: https://t.co/SWzpmYzIPq