Panopoly

Making Live hologram capture for #AR #VR #XR accessible to everyone!
Using Apple Lidar, Microsoft Kinects, and other cameras & AI

[Panopoly @panopolycapture]

https://t.co/FBw6QXhxBx
Some big web speed-ups; I think I can confidently say this playback is pretty smooth.

iphone SE doesn't even heat up any more.

[Panopoly @panopolycapture]

Re-post of last year's halloween pumpkin head capture

View it in browser here https://t.co/wE2WBLyqO9

[Panopoly @panopolycapture]

GUI for jetson&kinect combo almost back, just have to deal with some GLES2 restrictions...

[Panopoly @panopolycapture]

Kinda getting captures back to normal in web player.
But new format means new frag cleanup shaders. (ipad pro rear camera capture)

[Panopoly @panopolycapture]

Another benefit/reason to change output to mp4 is for previewing in the Files app

[Panopoly @panopolycapture]

Now outputting side by side/rainbow depth mp4's straight out of our capture app. (This is mac+kinect1, but same on IOS, jetson+kinectazure, pi+kinect1 etc)

Decoding here out of a <video> tag, pretty much just so I can get hardware accelleration on quest webxr & use webcodec

[Panopoly @panopolycapture]

Working on making Panopoly's output much more developer friendly;
MP4 output; instead of a more proprietry stream.
Rainbow depth, (instead of my whacky YUV packing), and side-by-side instead of multi-stream.

All for much easier integration, and allowing hardware decoding on web.

[Panopoly @panopolycapture]

A new version of the ios app is out on the public testflight
https://t.co/Yd4xxYRs8n

Mostly a lot of internal fixes & optimisations, only major thing left is that we're currently encoding a very compressed Z. (But colour video is back!)

[Panopoly @panopolycapture]

@jaroslavstehlik Some silly YUV encoding and then compressed to h264.
Not tied to h264, but we just have the widespread support & tools for it https://t.co/RNLSIYPUK9

[Panopoly @panopolycapture]

Kinect encoding & streaming to web. Nothing wild, just aligning all the renderers, encoders & decoders.

CMS upload is working too, so new ipad & iphone (truedepth & lidar both work) testflight beta shortly.

[Panopoly @panopolycapture]

@shlykur The capture app is cross platform (ios, mac, windows, pi, jetson, linux), currently supports a handful of cameras (built in ones, kinect 1,2 & azure)
The main player is web (with webxr), and a unity plugin

[Panopoly @panopolycapture]

I really need to remember to whack more #xr/#vr/#ar/#mr/#webxr/#volumetric #hologram #spatialcomputing hashtags on these things

[Panopoly @panopolycapture]

Also need to remember to put chroma back into the colour stream!

[Panopoly @panopolycapture]

...and ipad. Both currently missing the noise reduction/cleanup pass, but recording & streaming!

[Panopoly @panopolycapture]

Fixing some speed issues (ie. someone forgot they capped framerate to 30fps months ago)... mac + kinect1.

[Panopoly @panopolycapture]

Oh right, twitter compression.

[Panopoly @panopolycapture]

Mac & Kinect1 (green = noisy edge)

[Panopoly @panopolycapture]

@shlykur Couldn't really release the ios app without a 3D view :)
(plus now I can replay old files alongside live streams, and the device's view)

[Panopoly @panopolycapture]

Finally finished final minor issues, so now the web-renderer on https://t.co/e0jSErKjPq is running on the device whilst recording. (without noise reduc)
Not sure why colour is intermittent, but there's a lot of debug.
Same SwiftUI code as mac, same app .js as web/mac/win/ios etc!

[Panopoly @panopolycapture]

It also technically works on web, but currently I don't have any devices to hand which stream depth data to Web's MediaSources

[Panopoly @panopolycapture]

Our capture software has gone a bit more cross platform than planned...
Mac version now working (with kinect1 & 2), running off the same SwiftUI as ios, and same app code as ios, windows, pi, nvidia jetson