Shufan Sun

Research Assistant at UPenn CIS

[Shufan Sun @sofine_sun]

Hi Gul, yes—I think there’s definitely potential for these applications. Currently, our test set does not provide ground-truth real-world dimensions for the input images, so we have not yet quantitatively evaluated the accuracy of room dimensions, wall lengths, or object sizes.

At the moment, the initial room and object dimensions are estimated from the VLM’s common-sense priors, and object sizes are then fitted to the calibrated room dimensions using the corresponding scaled point cloud. If designers provide known dimensions, the reconstructed scene can be scaled accordingly, which we believe could enable more metrically faithful reconstructions.

Currently, our outputs are GLB files, and we’re working toward supporting formats that are more directly usable with professional architecture and design software.

[Shufan Sun @sofine_sun]

Excited to share our new 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 work, 𝗛𝗔𝗥𝗠𝗢𝗡𝗬, a framework for reconstructing compositional 3D scenes from single indoor images.

Much like a 3D artist, our VLM agent builds the scene in a 𝗵𝗶𝗲𝗿𝗮𝗿𝗰𝗵𝗶𝗰𝗮𝗹 𝗼𝗿𝗱𝗲𝗿: first the room structure and camera, then wall-mounted objects, free-standing furniture, and finally dependent decorations. After each stage, it compares its render against the input image, reflects on what went wrong, and iteratively corrects the scene to prevent errors from accumulating.

Even with older VLM backbones, our framework remains 𝗰𝗼𝗺𝗽𝗲𝘁𝗶𝘁𝗶𝘃𝗲 𝘄𝗶𝘁𝗵 𝗚𝗣𝗧-𝟲 𝗔𝘀𝘁𝗿𝗮 𝗶𝗻 𝘃𝗶𝘀𝘂𝗮𝗹 𝗾𝘂𝗮𝗹𝗶𝘁𝘆, 𝘄𝗵𝗶𝗹𝗲 𝗮𝗰𝗵𝗶𝗲𝘃𝗶𝗻𝗴 𝗯𝗲𝘁𝘁𝗲𝗿 𝗴𝗲𝗼𝗺𝗲𝘁𝗿𝗶𝗰 𝗮𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁 for single-image 3D scene reconstruction.

We also release 𝗛𝗔𝗥𝗠𝗢𝗡𝗬𝟯𝟬𝟬, a benchmark for single-image 3D scene reconstruction with 300 scenes across three difficulty tiers: easy, medium, and complicated.

🔗 Project: https://t.co/rxCKUzkHlD
📊 Leaderboard: https://t.co/viJWQyVHVu
🤗 Data: https://t.co/idDjT0Qz1r
📄 Paper: https://t.co/BqnnrUeFVy