@SemiAnalysis_ recently published a piece on where robotics companies are putting their inference.
Robot models are getting bigger, and on-robot compute isn't beefy enough to keep up. Jetson Thor is the top-of-market solution, but it has roughly a tenth of the FLOPs of a GB200. So companies are weaving between two approaches: either shrink the model to fit on the robot, or separate the "hard thinking" offboard.
They surveyed five different companies on how they did it:
๐ญ) @BostonDynamics
BD has split the stack. The visuomotor policy runs onboard on a Jetson Thor. The reasoning layer (which they say is similar to Gemini Robotics ER) runs in the cloud on Google TPUs. Their robot's tasks require sufficient generality, which means they need a frontier model. To make this split work, they've dedicated multiple teams entirely to networking.
๐ฎ) @AgilityRobotics
Agility keeps inference on their humanoid, Digit, on Jetson-class compute. The only decision-making that goes into the cloud is orchestration between robots. Agility runs on-board inference to meet customer requirements. Overcoming networking/IT constraints is hard at brownfield sites; off-board inference also raises safety concerns.
๐ฏ) @VerneRobotics
Verne runs everything locally on a Jetson or RTX 50-series. Verne bets that warehouse picking and packing doesn't need open-world reasoning, so a few-billion-parameter policy is sufficient.
๐ฐ) @SundayRobotics
Sunday was originally built for cloud inference, but when they deployed ACT-2 into homes, they quickly moved everything onboard within a few days. For them, latency was fine, but jitter killed them as the robot moved through natural dead spots inside a house. To have 99% accuracy, the robot simply can't handle that much jitter.
๐ฑ) @WeaveRobotics
Weave also runs inference on-board, but stays connected to the network anyway. Training data is constantly uploaded to the cloud, and teleoperators stand by to jump in if a robot fails a task.
The pattern I'm seeing, both in this post and across the industry, is that companies keep inference on-board, but the more general reasoning you need, the more impetus you have to split up your workload to use cloud compute.
Buy your next robot.
Reflex builds high-performance inference infrastructure for robotics and embodied AI. Now, separately from our inference platform, we sell the robots too.
https://t.co/n3FIVIWU0iโ
#Robotics#HumanoidRobots#EmbodiedAI#PhysicalAI#AI
This matters because a robot on shift intakes video whether anyone's watching or not, and every second of it has to be understood before the next action goes out.
Training gets paid once. Inference gets paid every day the robot is on the ground.
We're running Kimi-K3 on Spark at 490 tokens/s.
We partially matche images, so frames that barely changed don't get re-encoded. We publish that as a tenth of the compute behind a vision request.
A team is sending our cofounder and a robot to Mount Everest.
If physical AI is going to work everywhere, it should probably work at the top of the world too.
@dr8_unix come back alive with the robot.