Anton Matrosov

PlayCanvas • 3D Web Experiences • VR / AR Applications • Digital twins • 3DGS

[Anton Matrosov @antmatrosov]

If you ever wondered what it would feel like to look through your digital screen - I've created a small prototype of that!
https://t.co/eFPWCEOm1D

🔸Launch it on your laptop or tablet
🔸Start Head Tracking mode
🔸Enjoy!

#PlayCanvas #3DGS #GaussianSplatting

More details below👇

[Anton Matrosov @antmatrosov]

@bilawalsidhu Replicating that in 3DGS (real-world scan + synthetic lines) would actually be a cool application of the tech!

[Anton Matrosov @antmatrosov]

@bilawalsidhu Yep, I'm currently experimenting with something similar, but synthetic room with dynamic lighting👌 It might be finally a time to put 360 matterport tours to rest haha

[Anton Matrosov @antmatrosov]

@willeastcott @NASAMars Woah! this is why I'm paying for the internet haha

[Anton Matrosov @antmatrosov]

@Max_Eskandari Sure! I've spent about 1 week part time on this, and majority of time was spent on trying different motion reference shaders to make them work for Seedance. If I had to start from scratch now, I would probably spend ~1-2 days per scene. 3DGS training is just a couple hours

[Anton Matrosov @antmatrosov]

⚡️Hacking Seedance 2.0 for the Image to Gaussian Splat workflow

3DGS scene: https://t.co/hgCfmcwxLS
15sec video -> Postshot MCMC 1M 90k SSIM 0.94
Video renders: SuperSplat Editor

Details below👇

#3dgs #gaussiansplatting #seedance #aivideo #supersplat #playcanvas #lovecraft

[Anton Matrosov @antmatrosov]

@keitakuki That's so trippy! 😅

[Anton Matrosov @antmatrosov]

@lanamolx Runway, Huggsfield, Freepik - most popular ones atm

[Anton Matrosov @antmatrosov]

@lanamolx This is niche technical exercise to generate a small explorable 3D world from a single starting image. It's a hot research topic right now, so I've tried my hands at it as well 👌

[Anton Matrosov @antmatrosov]

@Max_Eskandari I'm pretty bad with 3D modelling, so I've used a simpler image-to-3DGS pipeline as the first iteration (https://t.co/q4k1fwc1pW), then designed my camera path there, applied position-to-color shader + placed dots planes manually, and it was done

[Anton Matrosov @antmatrosov]

@Max_Eskandari So you have to create a very crude 3D representation of your scene first -> then design a camera path -> render it into 15 sec video -> feed that into Seedance -> enjoy the result :)

[Anton Matrosov @antmatrosov]

@_trueuser_ Yep, that's correct 👌

[Anton Matrosov @antmatrosov]

And finally, you render your reference, pass it to the model, and hope and pray it will actually respect it and give you a somewhat decent result before you spend all you savings on this slot machine hehe [3/3]

[Anton Matrosov @antmatrosov]

🔸The most challenging part is preparing a motion reference video. First, you need to come up with some initial 3D representation of your scene, even before you have it in 3DGS. I've used ML Sharp for this + custom splat shader + dots overlay to improve motion tracking [1/3]

[Anton Matrosov @antmatrosov]

Then you design a camera path that covers your scene the best way possible, from all angles, in under 15 sec. Realistically you can squeeze in around ~30 distinct view points, and then connect them with some spline path [2/3]

[Anton Matrosov @antmatrosov]

🔸Seedance 2.0 is capped at 15 seconds, which gives you only about 360 frames to work with. For this task, you will need to use the Omni model, which can take both images and videos as input, along with a text prompt describing how those inputs should be used

[Anton Matrosov @antmatrosov]

@JulienReszka When I first saw results of my head tracking, I've asked myself the same thing!

[Anton Matrosov @antmatrosov]

If you ever wondered what it would feel like to look through your digital screen - I've created a small prototype of that!
https://t.co/eFPWCEOm1D

🔸Launch it on your laptop or tablet
🔸Start Head Tracking mode
🔸Enjoy!

#PlayCanvas #3DGS #GaussianSplatting

More details below👇

[Anton Matrosov @antmatrosov]

@the8thwall @playcanvas 4⃣: ?mode = all
Enable both World and Head Tracking on your device. By default Head Tracking is enabled only on laptops and tablets, and World Tracking - on tablets and mobiles

[Anton Matrosov @antmatrosov]

@the8thwall @playcanvas 2⃣: ?blur or ?faceblur = true | 0 ... 3. Enable it if you're shy and want to record a video, but with your face blurred out. Alternatively, you can always close face cam with a dedicated close button

3⃣: ?fs or ?fullscreen = true. Enable full screen (requires tap / click)