Hugues Bruyère

Try / Share / Iterate — Creative Technologist / Chief of Innovation @bonjourdpt

[Hugues Bruyère @smallfly]

Mixed Reality + Diffusion prototype as a tool for exploring concepts, styles, and moods by transforming real-world surroundings into alternate realities.

#MixedReality #MR #AI #StableDiffusion #Quest3

[Hugues Bruyère @smallfly]

@ImNeubert @XGRIDS_OFFICIAL @DJIGlobal 🙌😶‍🌫️

[Hugues Bruyère @smallfly]

I’ve been exploring what can be done with the developer data exported from @XGRIDS_OFFICIAL LCC Studio: combining the LiDAR data from a PortalCam capture with images captured separately, in this case using a @DJIGlobal Osmo 360.

Left: 3DGS generated directly in LCC Studio from the PortalCam capture.

Right: 3DGS trained in Spirula Studio (which is amazing) using the PortalCam LiDAR data for the geometry and the Osmo 360 images for the pixels.

Still a WIP... The hybrid version (trained with Spirula) has better colour and a different overall feel, while the LCC version is still noticeably sharper and preserves more fine detail.

Next test: adding some of the PortalCam’s own high-resolution images back into the Spirula dataset (the forward-facing rectilinear frames to start), while keeping the Osmo images as the majority of the dataset.

#GaussianSplatting #3DGS #PortalCam #SpirulaStudio #Osmo360

[Hugues Bruyère @smallfly]

@swyvl_io @XGRIDS_OFFICIAL @DJIGlobal Thanks! Happy to share what I’m learning along the way. More tests coming soon :)

[Hugues Bruyère @smallfly]

@dohae_metown @XGRIDS_OFFICIAL @DJIGlobal Thanks... My main goal is really to match LCC’s level of detail, or ideally surpass it 😶‍🌫️. The improved colour is definitely a plus, but some colour grading can always be done on a trained 3DGS. Fine details are much harder to recover!

[Hugues Bruyère @smallfly]

@Favour39388761 @XGRIDS_OFFICIAL @DJIGlobal Are you referring to potential alignment drift toward the ends of the capture, or distortion/alignment issues near the edges of the fisheye images?

[Hugues Bruyère @smallfly]

I’ve been exploring what can be done with the developer data exported from @XGRIDS_OFFICIAL LCC Studio: combining the LiDAR data from a PortalCam capture with images captured separately, in this case using a @DJIGlobal Osmo 360.

Left: 3DGS generated directly in LCC Studio from the PortalCam capture.

Right: 3DGS trained in Spirula Studio (which is amazing) using the PortalCam LiDAR data for the geometry and the Osmo 360 images for the pixels.

Still a WIP... The hybrid version (trained with Spirula) has better colour and a different overall feel, while the LCC version is still noticeably sharper and preserves more fine detail.

Next test: adding some of the PortalCam’s own high-resolution images back into the Spirula dataset (the forward-facing rectilinear frames to start), while keeping the Osmo images as the majority of the dataset.

#GaussianSplatting #3DGS #PortalCam #SpirulaStudio #Osmo360

[Hugues Bruyère @smallfly]

@debsahu @XGRIDS_OFFICIAL @DJIGlobal Oh, that’s really interesting!

[Hugues Bruyère @smallfly]

@debsahu @XGRIDS_OFFICIAL @DJIGlobal That would be great. I will try it. But shooting photos instead of video while capturing with the PortalCam would make the whole capture process a lot more complicated, at least for larger locations.

[Hugues Bruyère @smallfly]

@XGRIDS_OFFICIAL Thanks! 🙌😶‍🌫️ ... The hallway 3DGS used here was actually captured with the Portal Cam. Having a metric scale 3DGS from the PortalCam + LCC Studio makes registering it back to the physical space much easier.

[Hugues Bruyère @smallfly]

Vitre Experiment #4.

Winter is coming, so I brought a bit of spring into the hallway by adding synthetic 3DGS assets to the scene.

Still running smoothly.

#3DGS #GaussianSplatting #ComputerVision

[Hugues Bruyère @smallfly]

Vitre Experiment #4.

Winter is coming, so I brought a bit of spring into the hallway by adding synthetic 3DGS assets to the scene.

Still running smoothly.

#3DGS #GaussianSplatting #ComputerVision

[Hugues Bruyère @smallfly]

Finally got around to trying Spirula Studio, and I'm really impressed!

It simplifies the process of preparing datasets from 360° footage enormously.

Trained this 3D Gaussian Splatting scene from footage I captured back in 2022 with an Insta360 ONE RS 1-Inch 360.

#3DGS #GaussianSplatting #SpirulaStudio

[Hugues Bruyère @smallfly]

@greenie_guy The SfM part is a breeze compared to my previous workflow! Training takes about the same time, though.

Overall, this one took around 5 hours for 40–50k steps (can’t remember exactly) and ended up with about 25 million splats.

[Hugues Bruyère @smallfly]

Finally got around to trying Spirula Studio, and I'm really impressed!

It simplifies the process of preparing datasets from 360° footage enormously.

Trained this 3D Gaussian Splatting scene from footage I captured back in 2022 with an Insta360 ONE RS 1-Inch 360.

#3DGS #GaussianSplatting #SpirulaStudio

[Hugues Bruyère @smallfly]

I added a ToF sensor to the equation... an Femto Bolt.

Both the captured Gaussian Splat and the Femto Bolt depth data are registered to the physical space and rendered with the same off-axis projection… so now people passing through the hallway are captured and inserted into the 3DGS scene.

Still rough around the edges… but I like it.

This time I needed some help with the demo. Someone had to be the tracked POV while I strangely walked around the hallway :)

#3DGS #GaussianSplatting #ComputerVision

[Kyle McDonald @kcimc]

anyone who was around for the early 2010s kinect scene out was dreaming about this sorta thing. we just didn't have the fidelity and framerate yet. sooo satisfying to see this for real finally.

[Hugues Bruyère @smallfly]

@imcharleslo Not yet, at least. More work would need to be done to make this easily usable by someone else, along with some clear documentation.

[Hugues Bruyère @smallfly]

I added a ToF sensor to the equation... an Femto Bolt.

Both the captured Gaussian Splat and the Femto Bolt depth data are registered to the physical space and rendered with the same off-axis projection… so now people passing through the hallway are captured and inserted into the 3DGS scene.

Still rough around the edges… but I like it.

This time I needed some help with the demo. Someone had to be the tracked POV while I strangely walked around the hallway :)

#3DGS #GaussianSplatting #ComputerVision

[Hugues Bruyère @smallfly]

@TheKingKamil If you mean the trails behind the moving person, no long exposure or filter. That’s the live depth data from the Femto Bolt being rendered as a point cloud with a trail effect.

[Hugues Bruyère @smallfly]

@mrdoob @kcimc Oh, that would be interesting! I’ve actually been thinking about experimenting with Three.js for the rendering side. Curious to see how far I could push it performance-wise.