Daniel DeTone

Deep Nets and Geometry — what could go wrong?

[Daniel DeTone @ddetone]

@weikaih04 @allen_ai @uwcse @AIatMeta Very cool! Is this conditioning on the semi dense depth point cloud or RGB only?

[Daniel DeTone @ddetone]

The biggest impact of the Segment Anything line of work was not the actual image segmentation, but rather the flood of paper titles with the name “Any” in them. Cmon folks, let’s just call this generalization and move on!!

[Daniel DeTone @ddetone]

@MattNiessner I agree that metric 3D is critical! It’s a compressed, minimal representation. Playing devils advocate — humans also operate on projections of the 3D world and we are able to operate pretty well

[Daniel DeTone @ddetone]

New blog post about Boxer is live on the Project Aria website

[Project Aria @Meta @meta_aria]

How do you decompose a 2D image into accurate 3D object detections? You use🥊Boxer.

A new model from Reality Labs Research enables robust 3D object detection by "lifting" 2D proposals from off-the-shelf detectors like OWL-ViT and SAM into metric 3D space.

No more "flat" AI—this is about spatial intelligence for the next generation of wearables.

Blog🔗 https://t.co/WdOgFPzBBI
Website with links to download: https://t.co/HUdive9EYx

👉@ddetone

[Daniel DeTone @ddetone]

@neural_avb I was there in Barcelona! Epic

[Daniel DeTone @ddetone]

@Capsbrr Ah sorry, I thought you meant on Quest cameras, not running on the ML model on Quest hardware. I don’t think this model can run in realtime on Quest. Though it could probably be distilled significantly with further effort and maybe work

[Daniel DeTone @ddetone]

Today we release Boxer, a new lightweight approach that lifts open-world 2D bounding boxes to *metric* 3D: https://t.co/5IZ0tPlqvr

Here we show Boxer in action on an egocentric sequence captured from smart glasses:

[Daniel DeTone @ddetone]

@Capsbrr yes

[Daniel DeTone @ddetone]

@weikaih04 that would be great! we didn't train on much on outdoor data, I would expect a big boost the WildDet3D dataset training for outdoors

[Daniel DeTone @ddetone]

Cool showcase from @_satyam_ai running Boxer on RGB video using COLMAP for poses + pointcloud and GeoCalib for gravity estimation

[Satyam Kumar @_satyam_ai]

I implemented it and the ~8 degree gravity correction from GeoCalib made a real difference.

Look at the monitor - on the left (pose heuristic) the box is tilted and doesn't match the screen edges, on the right (GeoCalib) it wraps the monitor much more tightly. The shelf boxes at the top are also cleaner, less overshoot.

Yeah, the improvement is clear.

[Daniel DeTone @ddetone]

@_satyam_ai @pesarlin GeoCalib looking solid 💪

[Daniel DeTone @ddetone]

@_satyam_ai Amazing!

[Daniel DeTone @ddetone]

@_satyam_ai the gravity estimate looks a little bit off. another idea could be to run this per frame and take the global 3D average: https://t.co/KwRqnfa1F8

[Daniel DeTone @ddetone]

@yesitsarmin yes, the main limitation is the 2D detector here, but there are tons of better models (SAM3, VLMs) if you have the compute. for very cluttered scenes it doesn't work as well

[Daniel DeTone @ddetone]

@ElioenaiSiqCst Yes, I didn't show any examples of that but we trained on a massive internal-only Quest3 dataset

[Daniel DeTone @ddetone]

@BlueAquilae great question! I would not expect it to work well here, we would need to re-train it with a full 9 DoF representation. but feel free to try it out anyway, I'd be curious

[Daniel DeTone @ddetone]

@CleverBetTips The National?

[Daniel DeTone @ddetone]

@haodongli00 One limitation I found using both of those models is the runtime. For detecting 1000+ text prompts with SAM3 it takes 20+ sec per image. SAM3D also takes ~15 sec per object, so running on large datasets can be expensive. OWLv2 runs at ~30ms and Boxer takes ~20ms

[Daniel DeTone @ddetone]

@nickkarpov Feel free to file a GitHub issue if you have any problems! Will do my best to answer them quickly

[Daniel DeTone @ddetone]

For more details, check out the arxiv paper here: https://t.co/3CTK5TVDYc

[Daniel DeTone @ddetone]

BoxerNet runs FAST 🔥🔥, taking roughly 20ms on a 4090 with bfloat16 for ALL prompts in an image (e.g. 30 boxes in parallel)