stents

Unity Developer - Currently focused on AR/XR, creating interactive installations and other immersive experiences.

[stents @stents_dev]

Just wrapped up my #PICODevJam entry! Exploring @PICOXR's latest Sense and Interaction Pack features to create a treasure-hunting adventure in your room.

Join forces with an adorable mining sidekick and embark on a quest to unearth hidden gems in your home! Ready to explore? 💎

[stents @stents_dev]

@TracyTMacgrady @NianticLabs @the8thwall For this project I created a very simple API which just takes the camera image, and uses controlnet with monocular depth estimation as the control and sends back the result. So basically just camera image and prompt in, generated image out!

[stents @stents_dev]

Some recent experimentation with #StableDiffusion, #ControlNet and 8thWall for @NianticLabs & @the8thwall VPS Challenge

For technical details see the thread below! 🧵(1/4)

#MadeWith8thWall #AI #AR #AugmentedReality

[stents @stents_dev]

@stspanho @XRarchitect @Aidan_Wolf @BlockadeLabs True! Your work is amazing as well @stspanho!
The generative skybox idea definitely helped inspire my own project!
Would be awesome to combine the two ideas together 😃

[stents @stents_dev]

@Aidan_Wolf @BlockadeLabs Thanks @Aidan!
Awesome idea. A sketch mode is definitely something I want to explore in the future as well. Love @BlockadeLabs work!

[stents @stents_dev]

🧵(4/4) Summary

• Leverage ControlNet for image synthesis that fits any environment
• Utilize Niantic & 8th Wall's extensive 100k+ VPS database of locations and meshes to project onto
• Reimagining environments with the power of AI offers limitless creative possibilities!

[stents @stents_dev]

🧵(3/4) Projection

• Camera projection matrix is stored at the moment initial screenshot is captured
• Captured image is projected onto the VPS location's mesh, overlaying the camera feed in realtime
• Once a new image is generated, it is revealed, projected in the same way

[stents @stents_dev]

@Dilmerv Definitely not as good as the Quest hand tracking, as you mention, but hopefully they keep improving it. Happy to share more details later if you like.

[stents @stents_dev]

@Dilmerv @Dilmerv Pico 4 hand tracking worked ok for me when I tried it a couple of months back. They are lacking good samples, and I needed to create my own hacky hand interactor script, but you can see it in action near the end of this video: https://t.co/PlFKMmRS6h

[stents @stents_dev]

Passthrough AR test app on Pico 4. Including some hand interaction testing near the end (still needs some work).

Still on OS 5.1, and supposedly Pico OS 5.2 improves things significantly, so looking forward to testing that out soon.

#AR #VR #MR #questpro #PICO4

[stents @stents_dev]

Passthrough AR virtual lighting experiment. Running on Pico 4.

Passthrough Horror game in the making maybe?

#AR #VR #AugmentedReality #Pico4 #QuestPro

[stents @stents_dev]

@Dilmerv Pico 4 passthrough is definitely great for the price. I have been pretty happy with it so far.

Quest Pro passthrough is much more depth accurate though, due to the higher res IR cameras, allowing for overlaying color on the stereo IR feed. Pros and cons to each approach I guess.

[stents @stents_dev]

@Slime_BlaXun I do not really have any specific places, but YouTube tutorials are always a great reference for learning. Content creators like @Dilmerv have some great tutorials on XR Interaction Toolkit etc, mainly for Quest, but 95% of it can be done on Pico 4 as well.

[stents @stents_dev]

@dwarph Yes, the pico documentation is not great unfortunately, but it is possible using PXR_Boundary.EnableSeeThroughManual, which can be found in the API reference. Be sure to set the camera background alpha to 0 as well.

[stents @stents_dev]

Passthrough AR test app on Pico 4. Including some hand interaction testing near the end (still needs some work).

Still on OS 5.1, and supposedly Pico OS 5.2 improves things significantly, so looking forward to testing that out soon.

#AR #VR #MR #questpro #PICO4

[stents @stents_dev]

@Slime_BlaXun Currently Pico only support 1 play boundary at a time though, and recognition is not quite as reliable as on Quest... so hopefully they improve that in the future.

[stents @stents_dev]

@Slime_BlaXun Yes, there is no built in way of doing so yet as far as I know, so I built a custom scene management system to save occluders and objects relative to the Pico play boundary (which I also use to automatically set up colliders for the walls)

[stents @stents_dev]

@Slime_BlaXun But yes, Pico have their own SDK to enable AR Passthrough and for handling other things like controller vibration etc https://t.co/8N9sJZhOCm