This is the most interesting release I’ve seen this month!
AVM builds a 3D scene with no 3D at all. It generates a stream of views and remembers past ones by camera position, so you can walk away, come back, and the room still matches. No code yet.
Gave π⁴ a try on my own data today. It's impressive! This is from the opening scene of Spectre.
The only caveat, processing these 300 frames at 768x320 resolution took 37.5 GB of VRAM. You can downscale more, but the good results needs workstations GPUs.
Simulation is only as good as the person using it.
Prompting Astra or building a scene in MuJoCo is only the tip of the iceberg of what robotics sim can do.
Here's what I think you need to do to use it properly:
1. Calibrate your real robot first. Link masses, joint friction, motor response, latency. No two robots are the same even the same model. You should run joint sweeps and hard stops and see how yours actually behaves.
2. Don't hand-tune the sim. Run the same motions in sim and on hardware, score the difference and let a ML optimiser find the parameters. Past a few joints, hand-tuning doesn't work.
3. Know where physics is limited. Deformable objects, liquids, cables and messy contact won't match reality however you tune it. Train on real data and random physics instead to correct the simulation.
4. Give your policy memory if when randomising. Consistent friction and mass is necessary to maintain training history and understand/rationalise actions.
5. Something's wrong is the policy looks too good. Overfitting is common, policies will exploit physics when possible. For instance sliding contact or exceeding your motor torque. Check the sim.
For the startups that transfer sim2real, it's not about the quality of the simulator - rather the way they approach simulations rigorously.
Brett’s take on Chinese robotics is highly overgeneralized and I don’t think it’s good as it misleads the media. Not all Chinese robots are cheap. We just see a lot of the low cost ones. They have competitive humanoids coming out in 2027. Don’t sleep on China.
Brett Adock on Chinese humanoids:
"I think we care about something very different than what we think China cares about. China's almost in this race to show how capable their robots are at gymnastics.
We care about how do we ship the robots at home with zero-shot, long-horizon reasoning. And you're not seeing that out of China. These groups, even like Unitree, they don't build good hardware when you buy it. They're super tiny. They come out of the box, they wave, and with the controller, you're like, what else do I do with it? It's unclear.
They have no hands. And they have compute the size of a USB stick. So you basically can't run any real models on them. Seriously, all the groups that are doing research put compute backpacks on them. And the sensors aren't great. The hardware was basically duct-taped together. It wasn't manufactured well.
But will they build better robots over time? Certainly. They're going to build, I think over time, probably great robots. We just got to go beat them at this. It's a real race here."
It's been a while since I scanned anything for 3DGS. Here is a 2 minute scan of a bunch of trees by my house. I didn't focus on any one tree so I think the quality of detail isn't great. Fun getting back at it though!
Checking out the excellent demo for Solv3R! I sped up the reconstruction 2X and the 3DGS processed in about a minute. Next up, trying my own data and hacking mods on the demo.
@taiyasaki amazing work! Wished all projects shipped with this level of a demo.
@Sentdex Perhaps it got the reward function wrong.
Next time the prompt should be: “route all planes to be as efficient as possible and completing their routes. Make no mistakes”