harpreet

hacker-in-residence/ principal devrel engineer. data curation for physical and visual ai systems. in my physical ai era

[harpreet @DataScienceHarp]

a robot can recognize every object in a room and still not know how to open a single drawer

knowing there's a cabinet is scene understanding. knowing which handle to grab, which direction to pull, and that it's a drawer not a door is affordance understanding

almost no dataset has it in 3D

SceneFun3D: 14,800+ functional elements across 710 laser-scanned indoor scenes. every handle, knob, button, and switch labeled with its affordance, motion axis, motion type, and a natural language task description

each scene's 3D point cloud with its iPad video recordings in fiftyone. the affordance annotations live in 3D and project into the video frames

check it out here https://t.co/JwG6x31RmJ

[LeRobot @LeRobotHF]

New in LeRobot: DM0.5 by @Dexmal_AI 🤖

A 4B VLM with a 680M flow-matching action expert, pretrained on 7 robot embodiments. Right now it's the top open-source model on RoboColiseum.

Fine-tune it on your robot:
--policy.path=lerobot/dm05_base
Docs 👇
https://t.co/7OqgQyLY4W

[harpreet @DataScienceHarp]

a robot can cross a hallway without hitting anyone and still make everyone in it uncomfortable

collision-free and socially acceptable are different problems, and almost all navigation data only measures the first

NavWareSet records the second

NavWareSet records both. seven social navigation scenarios (frontal approach, blind corner, following, perpendicular crossing), each run twice under matched conditions: once with socially compliant behavior, once without. same room, same people, same starting positions.

the only variable is whether the robot navigates like it knows humans have personal space

robot onboard lidar and camera plus an overhead ground truth station tracking every pedestrian in 3D across the full episode

loaded as native mcap in fiftyone. scrub the robot's camera, both lidar streams, and the annotated pedestrian trajectories on one synced timeline.

filter by scenario and behavior to compare compliant vs non-compliant side by side

start here, read the dataset card: https://t.co/Xkj9SWj6Du

#mcap #skills #computervision #ai #artificialintelligence #machinevision #machinelearning #physicalai

[srinivijay @srini_vij]

Fifteen years ago, doing SLAM with a single camera didn't even stand a chance.

I was working on SLAM back then. Point Cloud Library had just been released. ROS1 was picking up steam. And if you wanted good accuracy, you used a LiDAR of some sort.

Cameras alone didn't cut it. Not even stereo. Bring up a single camera and nobody took it seriously.

Then I came across this open-source model rebuilding 3D scenes from one camera, live, at around 20 frames per second, and staying stable over long sequences.

That still blows my mind.

And it isn't only perception and mapping that grew up. Position estimation came a really long way too. Factor graphs, IMU preintegration and a stack of other techniques each pushed the field forward in their own way.

If you're a robotics nerd, you know how exciting this is.

The setup that didn't stand a chance in my SLAM days is now code you can clone and run.

[harpreet @DataScienceHarp]

the spray behind a truck on a wet highway is solid to lidar and invisible to radar

your camera sees a blur, your lidar sees a wall of false objects, and the disagreement is exactly the signal

SemanticSpray++ from Ulm / BMW: 36 vehicle-following episodes on a closed wet airstrip, 50-130 km/h, with per-point semantic labels on both lidar and radar telling you exactly which returns are spray noise and which are the actual vehicle. plus 2D camera boxes and 3D lidar boxes on every frame

loaded as native mcap in fiftyone so you can scrub camera, lidar, and radar together and watch the spray noise light up in the point cloud while the boxes track the lead vehicle through it

checkout the dataset here: https://t.co/vZapSnAaZv

or get hands on with this hugging face space: https://t.co/m9Gai2R9nW

#mcap #skills #computervision #ai #artificialintelligence #machinevision #machinelearning #physicalai

[harpreet @DataScienceHarp]

seven cameras and a LiDAR on Indian city streets. no boxes in the files.

this Minus Zero slice is 8 MCAP files, about 93 GB. cameras: front, front-left, front-right, front-tilted, rear, rear-left, rear-right. LiDAR is stored as autonome.LidarFrame protobuf, with GNSS and pose. video is H.265. recorded October 2025. CC BY-NC 4.0. a 1,469-file collection, about 3.2 TB, is the full set.

https://t.co/AtyH2YJKIp

#PhysicalAI #AutonomousDriving

[harpreet @DataScienceHarp]

species labels at this count are rare in a vision set.

TreeOfLife-200M holds 233M images across 933K taxa, labeled from kingdom through species.

https://t.co/gLkDYFFU3H

#PhysicalAI #BioML

[harpreet @DataScienceHarp]

lidar sees a blizzard as ten thousand small objects. radar sees nothing at all

most driving datasets are shot on sunny days in california.

Boreas is UTIAS's answer: 128-beam lidar, 360 degree radar, and 5MP camera driving the same Toronto route for a full year through sun, rain, and snow.

326,180 3D boxes for cars, pedestrians, and cyclists across 7,111 labeled frames

loaded as native mcap in fiftyone so you can scrub camera, lidar, and radar on one synced timeline, and watch the 3D boxes render live on the point cloud and project onto the camera and radar images

checkout the dataset here: https://t.co/KivCRtDndW

or get hands-on with this hugging face space: https://t.co/7WyTZUPj2d

#mcap #skills #computervision #ai #artificialintelligence #machinevision #machinelearning #physicalai

[Suvaditya Mukherjee @halcyonrayes]

new day, new dataset 🤗!

we now have 900+ 3D files from @NASA on the hub to explore!

- includes space shuttles, satellites, and more! 🚀
- permissively licensed, filtered

dataset link: https://t.co/ykroTZpgz2

use this space to explore and look around: https://t.co/aQsUql0moW

[harpreet @DataScienceHarp]

closed-loop driving success on AlpaSim is up 33 percentage points.

LaPla is a VQ-VAE latent-aligned vision-language-action planner. it predicts trajectories on multi-view driving video and reports L2 trajectory error.

https://t.co/YsuBz7vysn

#PhysicalAI #AutonomousDriving

[harpreet @DataScienceHarp]

evaluating world collection usually runs into the same bottleneck:
clean lab setups rarely match messy physical environments.

World Collection captures the ground truth:

42 mathematically generated gazebo simulation worlds with programmatic python sdf generation and standardized start poses for navigation benchmarks.

https://t.co/uhhCRRk0it

[harpreet @DataScienceHarp]

the contour is where a segmentation model gets sure and wrong.

RABR reads pixel-wise segmentation margin entropy and test-time augmentation variance along boundary contours to isolate overconfident failure regions.

https://t.co/RTdG1oMutl

#PhysicalAI #ComputerVision

[harpreet @DataScienceHarp]

@mervenoyann @huggingface im a simple man, i see a merve post i smash a like

[harpreet @DataScienceHarp]

@giffmana oh bro that’s so cold

“Professional echo chamber inhabitant…” 😂

[harpreet @DataScienceHarp]

@andrewdsouza @boardyai @illscience ngl, im a fan. @boardyai is my guy!

[harpreet @DataScienceHarp]

@ptruiz_dev Winnipeg, so def a very very different climate

[harpreet @DataScienceHarp]

@WuSheraton92600 Thank you!!

[harpreet @DataScienceHarp]

a static avatar still has to move on command.

AvatarDynamizer turns a static 3D avatar into a controllable 4D one with generative dynamic textures in texture space, and it includes a new large multi-view set. outputs are 3D gaussians and dynamic texture maps.

https://t.co/DyNwvpGhiY

#PhysicalAI #ComputerVision

[harpreet @DataScienceHarp]

the mask matches the pixel, on millions of composed images.

Synthetic Object Compositions has 2M images (FC-1M and GC-1M) and 20M synthetic segments across more than 47,000 categories, with IC-Light relighting and pixel-perfect masks. labels include panoptic masks, bounding boxes, referring expressions, and categories.

https://t.co/gFoyU2ryBp

#PhysicalAI #ComputerVision

[harpreet @DataScienceHarp]

two of voxel51's finest, don't miss it

[Jimmy at Voxel51 @jimmy_voxel51]

Join us Nov 9 for the "Agentic Labeling for Specialized Visual Data" workshop! Register for the Zoom - https://t.co/WrZCrRf7p9

Frontier multimodal models handle familiar concepts well and struggle on niche ones: sorting drill-core samples into 35 geological classes, or telling adhesion from scarring on a ceramic substrate. Voxel51's study across 11 specialized visual datasets in manufacturing inspection, geology, and medical imaging found that a few labeled examples in the prompt can close much of that gap, and it also found where that stops working.

This workshop is about applying those results to your own labeling work. We start from what the study measured, then build a few-shot labeling task live with Agentic Labeling in @Voxel51 on a real inspection dataset, and use the data to decide when to keep prompting and when to train a specialist model.

Run it on your own data! Attendees get free GPU compute from @nebiusai to try the workflow after the session.

#Voxel51 #FiftyOne #MultimodalAI #ComputerVision #MachineLearning #AIResearch