PentaCoxian

animated 4DGS/standalone mobile VR 3DGS/multi view 3D pose estimation/temporally stable video depth estimation
画像/映像系の色々をやってます。

[PentaCoxian @pentacoxian_dev]

品質はまだまだだけどフルスクラッチ・自作の商用利用可能な4DGSシステム出来てきた。3DGS連番のパラパラ漫画方式じゃないし、RTX4090レベルでトレーニング可能。
比較しやすいように試しによく使われてる4dvのcorgiデータでも試した。(自作の撮影の方では36台のXperiaを使ってる)
#4dgs

[PentaCoxian @pentacoxian_dev]

@_ryu15_ まぁ完全に動かせないかと言われるとそうでもないんですが、わざわざこのモデルファイル選ばんでも…って感じの気持ちになっちゃいますねw

[PentaCoxian @pentacoxian_dev]

ちょっと普段のポスト内容とは違いますが、最近話題の(?)V100でローカルLLMを検証したので記事公開します。Qiitaにも同一内容で投稿してます。 https://t.co/cgJHdSOWj2

[PentaCoxian @pentacoxian_dev]

I'd assume that'll work fine, although for me, using QZSS CLAS(Japanese L6 RTK) compatable device like the mosaic-go CLAS is more attractive. I guess it'll depend on where you live and the RTK solutions provided in that area.
Found a pretty cheap and slim CLAS compatable helix antenna so I might give it a try in the future.

[PentaCoxian @pentacoxian_dev]

Osmo360 初代、ガウシアン用魔改造pentacoxianモデル完成！

120MPモードだとステッチされて出力される問題の解決と通常3〜4秒間隔でしか撮影出来なかったのを2秒に短縮。7680^2 の全周魚眼2枚で保存されるのでシームライン気にせず再構築できる。

最後のロゴはお遊びw
#3dgs

[PentaCoxian @pentacoxian_dev]

GoPro ILSの16K VR180撮影用リグできた。
近いうちに撮影サンプル上げられるといいな。
#VR180 #SpatialVideo

[PentaCoxian @pentacoxian_dev]

えーと届きました。まさか今更GoPro買うことになるとは…

リグこれから作るから撮影少し先だけど、なんちゃって16K VR180…？(縦が行かない)

Cine immersive の20分の1の値段でどこまで撮れるか比較したいね。
#VR180

[PentaCoxian @pentacoxian_dev]

Osmo360 初代にusb経由でGPS RTK情報埋め込みできるファーム作ったけど3DGSのスケール調整とかマルチカメラの時刻同期に需要あるかなぁ

日本だとQZSSのCLAS対応モジュールあれば単体でRTK補正できるはずだけど…

[PentaCoxian @pentacoxian_dev]

As long as the RTK module can send standard NMEA over USB, I can make the osmo embed the RTK info on the taken pictures exif data directly from the module connected to the osmo USB port so just a standard RTK module that can connect over USB is enough.

I don't have a good RTK module at hand so used a esp32 emulating NMEA packets over USB and got the osmo to embed the simulated info into exif.

[PentaCoxian @pentacoxian_dev]

@kaikau It might work but processing on the iPhone was much easier. If I stitch on device, even if it did work, it'll probably drain the battery super fast so I didn't see the point trying.

[PentaCoxian @pentacoxian_dev]

As long as the RTK module can send standard NMEA over USB, I can make the osmo embed the RTK info on the taken pictures exif data directly from the module connected to the osmo USB port so just a standard RTK module that can connect over USB is enough.

I don't have a good RTK module at hand so used a esp32 emulating NMEA packets over USB and got the osmo to embed the simulated info into exif.

[PentaCoxian @pentacoxian_dev]

Yeah, the new zcam seems really nice, especially the poe will clean up wiring for multicam rigs.

The price estimate of 1kusd is attractive but I've had past problems of getting large quantities of zcam bodies and locally the rx0m2 combo prices will probably be like half the cost of the e1n due to the yen value sucking right now so I'll probably have to stick to Sony :(

Hope your multicam car project goes well in any case!

[PentaCoxian @pentacoxian_dev]

Oh just realized you from the LFS Discord server!
Posted there too a few hours ago but have you considered the rx0m2 from Sony? It doesn't have global shutter or genlock but it does have quite good per frame network sync using control boxes. Availability is getting spotty but should be still within or around 1000usd per camera+control box depending on where you live.

Used it for a 4dgs rig and was quite good, especially the MFT mount converted ones.

[PentaCoxian @pentacoxian_dev]

Oof, I heard getting dji cameras in the US is quite hard right now due to the dji ban.
The fuzzyness kind of sounds like a calibration error so I hope a new calibration will help. The LichtFeld Studio discord server has a channel on camera pose estimation so might be worth visiting.

[PentaCoxian @pentacoxian_dev]

Agreed, 1sec is too sparse for some timelapse applications and that's why my app has multicam support :)

The 120MP capture can be shot at the minimum spacing 1.5sec apart so say if I connect 3 Osmo360s then set them to shoot 0.5sec apart from each other I get a 0.5 fps 120mp timelapse!

[PentaCoxian @pentacoxian_dev]

Just for fun, made a iOS based osmo dual lens stream intercept & stich/encoder for YouTube realtime streaming and got 4k60 panorama video streaming(YouTube livestream resolution limitation) with stitching done on the iPhone 15 pro in realtime!

Making a stitching algo with good quality and speed was a bit of a hassle but I think for a uncalibrated lens distortion based realtime stitcher I got it quite good (you can still see a faint seam near the ceiling)

[Ka'i Kau @kaikau]

@pentacoxian_dev The other fun part is being able to stream to YouTube, but the resolution is very low.

[PentaCoxian @pentacoxian_dev]

The Osmo360 first Gen can only take 8k in video so that's the main reason for switching to 120MP stills for me. The extra resolution does help, and with a minimum 1.5sec interval, it's almost the same interval I set when extracting from video anyways...

And yes for ground based scanning I do mask myself out, although recently I've been more impressed with the mavic 4pro 32K photo panoramas (don't need fisheye adopted masking models for the camera operator masking)

[PentaCoxian @pentacoxian_dev]

Done on the dji mavic 4pro and LFS.
Thanks to @lichtfeldstudio and @janusch_patas for building such a good tool.

Not even on the latest beta and the backgrounds still look very good. Can't wait rerun on the latest nightly.

[PentaCoxian @pentacoxian_dev]

@TokyoWarfare The faster capture only works with a firmware modified Osmo360 first Gen. If you mean the 30MP panoramic photo ( I don't recall a 24MP, sorry if you're talking about a different mode) the delay is about 1sec interval with the modified osmo.

[PentaCoxian @pentacoxian_dev]

yeah, the security quality on these hardware is pretty loose and the modifications it enables is pretty fun ;)

As for streaming, if you're talking about panoramic video, you actually can get the dual 4k Fisheyes to stream video over wifi, although you'd need a good realtime 360 stitcher software to get the 8k panorama video (Also Japan sadly limits the use of 5GHz wifi so I can't push anything near 100Mb/s which limits the maximum fps at 8k)

[PentaCoxian @pentacoxian_dev]

現実映像のマルチカメラからの3Dポーズ推定のほうが専門ではあるんですけどアニメ調の3Dvtuberぽい映像から3Dポーズ推定は可能なのかちょっと試してみた。

普通の人間用のポーズモデルだとドメイン違いすぎてまともに使えないので半自作モデルです。

いま手元に自由に公開できる3DvtuberのモーションがなかったのでAI生成の映像で改善点は大量にあるものの、歩行/指含め想像より綺麗に立体ポーズ出せてる。

[PentaCoxian @pentacoxian_dev]

Will be working on the Osmo360 ii later, although I don't know how much the firmware tweaks for the dual lens fast 120MP capture would work on the ii.

The remote 120MP capture with gyro data does work if you just want to avoid the 16K timlapse video mode on the Osmo360 ii.

Haven't considered the insta 360 x6, which features are you looking for?

[PentaCoxian @pentacoxian_dev]

いや、光源の計算さすがに時間かかりすぎじゃね？と思ったら光源位置変えてなくても毎フレーム再計算走ってたw
さすがに意味ないので高速化

[PentaCoxian @pentacoxian_dev]

日本刀の3DGSプロトタイプ作りました！
OLATを参考にgsplat等で書き直したinriaなどに依存しないNCライセンスフリーのRelightable 3DGSです。まだ光源の計算に時間かかってしまうのが改善点ですが、自由に光源移動・視点移動でき、光の反射や角度による刃紋などの見え方の違いが見れます〜 #3dgs