Last Friday we threw a party at our SF office and let the robots host.
The bartender was in Armenia, where it was 7 am Another robot worked the room, controlled from Fresno.
When the Pome-Grasp-It hit the ceiling, a robot cleaned it up.
Last Friday we threw a party at our SF office and let the robots host.
The bartender was in Armenia, where it was 7 am Another robot worked the room, controlled from Fresno.
When the Pome-Grasp-It hit the ceiling, a robot cleaned it up.
Most teleoperation is one person, one robot. On the way to autonomy one person has to watch many. So we put a few of ours on one page: @i2rt_robotics arms, a @UnitreeRobotics Go2, a @FrodoBots Earth Rover and a @pollenrobotics Reachy Mini, and had one person watch all of them.
The ratio moves the other way too: many people, or agents, able to drop into the same robot and hand it between them. In the video we join the Go2, ask the person driving it for control, get it, and walk it around the office.
Check it out on https://t.co/5fc5x9BxKs
Media over QUIC is not limited to media.
@wrangelvid of Evolve Robotics will explore what comes next in โMoQ beyond mediaโ at DEMOQED.
October 8 in San Francisco: https://t.co/R7MlvB81SG
Media over QUIC is not limited to media.
@wrangelvid of Evolve Robotics will explore what comes next in โMoQ beyond mediaโ at DEMOQED.
October 8 in San Francisco: https://t.co/R7MlvB81SG
Getting NVIDIA's scenes into a browser was the bigger job. The forklift alone comes with over 400 MB of textures. Models upload once to our asset cloud, get optimized for the operator's device and stay cached. Same for Drake and MuJoCo.
We connected @NVIDIARobotics Isaac Lab, so people can give demonstrations in Isaac simulations from anywhere.
In the video we drove NVIDIA's forklift through their full warehouse from a browser with a gamepad, picked up a pallet and dropped it off in an aisle.
Simulation is a cheap way to collect demonstrations without waiting on hardware. Isaac Lab already ships teleoperation for its tasks, but the headset has to be on the same network as the simulation. With our integration the operator can be anywhere.
The operator's side works the way it did on last week's arms. This one runs in Drake, and we also have integrations for MuJoCo and Isaac Lab.
Try it in simulation first: https://t.co/5fc5x9BxKs
Last week's setup cost $8.5k. This one cost nothing.
Two arms pack boxes into a bin, both simulated. Same headset and controllers as last week, and a pulse each time a gripper closes on a box. No arms, no cameras, no Jetson.
It's the cheap place to try things. A new grasp or controller change gets tested before it goes near a real arm. A new operator learns the task without breaking anything. And a simulated bench resets in a second, so demonstrations don't wait on hardware.
@mikisafr@wessam_hamid We had a really similar experience. With the right visual feedback it works pretty well and unlocks scale for remote intervention.
Especially noticeable for robots with more degrees of freedom like the franka or kuka arms.
We put two robot arms on a bench and sent a towel-folding task to our operator network.
An operator in a headset drove the arms with VR controllers. Minutes later the towel was folded. They were never in the room.
Arms @i2rt_robotics, cameras @Stereolabs3D, compute @NVIDIA
Seeing contact on video and deciding to stop takes a moment. A buzz in your hand is faster, and enough to ease off before you ram the table.
Past a force threshold you feel a pulse in the matching controller and see an arrow. If you pay attention youโll see the arrows!
While your hand moves freely, robot arms got limits. You need to see when it can't follow.
When requested and actual gripper poses drift apart, a translucent gripper fades in where you asked it to go, on the live video. Augmented reality inside VR.
Watch the first few seconds.
Though we tried voxelized depth cameras first, which was fine for big objects, but discretization error made thin things like a towel edge impossible to judge.
Folding a towel remotely means judging depth and moving the arm where you mean to.
Stereo, one camera per eye, lets you judge the gap between gripper and towel.
Two VR controllers drive the arms through differential IK, without leader arms.