Shadnam Khan

Building @multiset_ai - Visual positioning for Physical AI | Human & machine perception | Mapping, localization, glasses, robots

[Shadnam Khan @ShadnamK]

Every system in a plant has an owner. The ERP, the maintenance system, the twin, the robots.

Nobody owns the layer that ties any of them to the building, so it gets bought inside someone else's project and dies when that project closes. @multiset_ai

[Shadnam Khan @ShadnamK]

@RiadRaphael @multiset_ai Good question. Two ways:
1. Prior map with new version - the section that had drastic changes are re-mapped and not the entire facility - https://t.co/eudsFctrRe
2. Localizations and images captured contribute to map updates (with opt-in)

[Shadnam Khan @ShadnamK]

Hundreds of photos get taken on a site every week and almost none of them know where they were taken.

This one does. It's sitting at the exact spot and angle it was shot from, inside a scan of the floor, with the confidence on screen. @multiset_ai

[Shadnam Khan @ShadnamK]

@emmettlawlor_ @multiset_ai We use a prior map from more than 10+ file formats (e57, 3dgs, 360 video, usdz, spz, etc.) and localize the images. Where possible, taking any available assistance from IMU, GNSS, etc.

[Shadnam Khan @ShadnamK]

@ThisMightWrk @multiset_ai Oh yea! I pulled photos from Google image search and that's the results you are seeing.

@bilawalsidhu actually built a prototype of the spatial memory concept on @multiset_ai VPS: https://t.co/3kOJWzkI6M

[Bilawal Sidhu @bilawalsidhu]

What if you could see every moment all at once?

I turned a mama deer and her babies walk into a 4D reconstruction -- and the internet got philosophical.

But the deer is just the beginning. Welcome to the geospatial memory palace.

Full deep dive: https://t.co/oPVtpcuhvW

[Shadnam Khan @ShadnamK]

Crowd-sourced photos of Westminster, each one dropped back at the exact spot and angle it was taken from.

We didn't scan anything. Testing on Google's 3D Tiles. @multiset_ai

[Shadnam Khan @ShadnamK]

The green figure is standing inside the house. The camera never goes in.

Anchor something in a room and it stays put whether or not the camera can see that room.

The map holds the whole building. The camera only ever sees one room of it.

Shot by @bilawalsidhu @multiset_ai

[Shadnam Khan @ShadnamK]

I ran out of time at #ROSCon Lighting Talks before I got to the ask.

This is what I was showing. A Unitree Go2, one camera frame in, absolute pose out, against a scan the building already had.

Open source, Apache-2.0. Repo in the reply. @multiset_ai

[Shadnam Khan @ShadnamK]

@m_saharia @zomato @deepigoyal The weather sensors are a clever find. How often does the LiDAR base get refreshed?

[Shadnam Khan @ShadnamK]

@Alacritic_Super No prebuilt map is the interesting choice. What happens on the second flight through the same tunnel, when it has to agree with yesterday's map?

[Shadnam Khan @ShadnamK]

@willeastcott @playcanvas That's a lovely use for it. I keep wondering what it would take for a visitor standing in the real room to see the same hotspots on the same canvases.

[Shadnam Khan @ShadnamK]

@smallfly @kcimc Makes sense. Is it latency or jitter that breaks the illusion first?

[Shadnam Khan @ShadnamK]

@takesenit Agreed, a map that already knows where it sits in the world skips a whole alignment step. Who's buying the K2 in Japan so far, survey firms or newer teams?

[Shadnam Khan @ShadnamK]

Every plant has a system that knows exactly which valve needs work, and no way to get that answer to the person standing in front of it.

The work order has known since someone typed it in. It just can't get down the stairs.

At a desk it pays back once. At the valve it pays on every round, every shutdown, every handover. @multiset_ai

[Shadnam Khan @ShadnamK]

@engineers_feed The open piece is that every one of them has to agree where it is inside a plant whose layout changes monthly, and the robot budget tends to get approved before the map budget.

[Shadnam Khan @ShadnamK]

@NOELreports Both sides of the war shipped optical positioning in the same week, which is the clearest signal yet that satellite navigation is now treated as optional. Wondering what that approach does indoors, where there's no terrain underneath to match against.

[Shadnam Khan @ShadnamK]

@spatialinsider Real validation for spatial intelligence, and the price says so. Supplying the coordinate frame that ties a generated world to the actual building a robot has to stand in is up next.

[Shadnam Khan @ShadnamK]

@lukas_m_ziegler Awesome work! SLAM gives me a map that's consistent with itself, and I've spent whole days learning that this is not consistent with the building.