matthias heyrman

[matthias heyrman @MatthiasCaro]

What does a humanoid need to look at to cross challenging terrain?

People only look around their next few planned steps. So we showed a humanoid only that.

Watch Your Step: Anticipated Perception for Humanoid Locomotion

🌐 https://t.co/W4NNjxI0tX
📄 https://t.co/wDOEsxkYdj

[matthias heyrman @MatthiasCaro]

@hyhieu226 The humble MLP will always have its place.

[Junzhe (JJ) He @JayHe748646]

Tty ACP yourself!

https://t.co/l3VF2zPUVx

[matthias heyrman @MatthiasCaro]

Amazing demonstration of generalized locomotion skills from @JayHe748646 @ChongZzZhang. The simulation demo is great, and it looks even better on hardware. This really pushes the capabilities of humanoid robots, and everyone should check it out as a new "must read" work.

[C. Zhang @ChongZzZhang]

https://t.co/cqYb3N9e85

ACP yourself.
Video 1: vault, climbing, jumping, get up,
Video 2: sit
Video 3: ultra-hard standing on vaulting wall

PC & Phone both supported.

(more scenes still under construction)

[matthias heyrman @MatthiasCaro]

@SreeramG Robotics becomes more accessible every day. I do think that the struggle to transfer assistance curriculums due to the shifting transition function is a flaw in the implementation as is, but the speed of training (and iteration) means that we'll see quick progress.

[matthias heyrman @MatthiasCaro]

Trained the FPM + locomotion policy inside a world model, SuperMimic-style, on my 4070 Super. Far from perfect, the force assistance curriculum didn't carry over, but sparse terrain success surprised me. Not sure about deploying such a policy, but the direction is very promising.

[Manthan Patel @patelm99]

Vision foundation models like DINO and SigLip now power much of robot learning.

But they are built for RGB. What about depth?

Introducing DeFM: a foundation model for depth images, built for robotics.

🌐 https://t.co/KzloKyRZyF

Weights, code, and demo are out today 🧵

[matthias heyrman @MatthiasCaro]

Someone please add different robots with a stats screen and release on Steam / Switch.

[Chen Tessler @ChenTessler]

https://t.co/TIv2bE6OjS

In this racing game, a simulated G1 is controlled using the exact same robust controller that runs on the real robot.
No privileged information.

See if you can win the race! 🏃🏻‍♂️

[Lokesh Krishna @lok_s_h]

orcs v0.1.1 is out 🚀 https://t.co/6S5ABRub6v

🧗 perceptive whole-body locomotion task on EgoHTR: one adapted policy for rocky climbs and window crossing

🛠️ agentic UMR retargeting skill: leave frame-wise notes; your agent revises the reference

[matthias heyrman @MatthiasCaro]

@Rawlala1 Yes, but the level is not as high and the behaviour is unusual for the reward set. The world model struggles to adapt to curricula that change the state transitions (like the vertical force assistance) which typically improve the terrain uptake. Hard to learn a moving target.

[matthias heyrman @MatthiasCaro]

@srigopal @leggedrobotics We test the policy on several out of distribution terrains, such as wide gaps, drops, and high amplitude noise, and we see that the predictions hold up. Since the policy observes the terrain around the predictions, it can react in mid-air and change its step to a better location.

[matthias heyrman @MatthiasCaro]

@EpisodeYang Would love to see it! If over the weekend I can get something more stable, I wonder how the sim2real performance is.

[matthias heyrman @MatthiasCaro]

@0xkubj Since there seems to be interest in this, I'll see if I can refine it a bit more before adding on GitHub. In the meantime, the training code for the original work (trained in sim) is already available: https://t.co/aEAWWflakZ

[matthias heyrman @MatthiasCaro]

@realmyzz We don't use a pretrained world model. As in SuperMimic, the model is trained from scratch on the states that the locomotion policy visits in sim rollouts, and the policy is then improved by backpropagating through imagined rollouts of the world model.

[matthias heyrman @MatthiasCaro]

@ZappyZappy7 Thanks for sharing our work! You can try the policy and foothold predictor yourself in your browser on the project page, or see the full implementation on GitHub: https://t.co/aEAWWflakZ

More results in our thread: https://t.co/HhojzLoRBX

[matthias heyrman @MatthiasCaro]

What does a humanoid need to look at to cross challenging terrain?

People only look around their next few planned steps. So we showed a humanoid only that.

Watch Your Step: Anticipated Perception for Humanoid Locomotion

🌐 https://t.co/W4NNjxI0tX
📄 https://t.co/wDOEsxkYdj

[matthias heyrman @MatthiasCaro]

@srigopal @leggedrobotics The foothold predictor's output is based only on the previous step of proprioceptive and velocity command input, so there is no compounding error.

[Robotic Systems Lab @leggedrobotics]

Can a humanoid learn faster and better by seeing less?

We train a small model to predict the policy's next steps, and show the policy only the terrain there. Together, they learn to cross gaps, stairs, drops and boxes.

🌐 https://t.co/lgDDkoJBrp
📄 https://t.co/qYZ8QZLdCy

[matthias heyrman @MatthiasCaro]

@NFT_Chen Time to turn my home pc with a 4070 into a dedicated training rig.

[matthias heyrman @MatthiasCaro]

@Ronald_vanLoon @spaceandtech_ The first? Anymal on wheels/swissmile (now RIVR) did this like 5 years ago and it's been done and highly publicised many times since.

[matthias heyrman @MatthiasCaro]

@VaderResearch I doubt that the cost of retrofitting hardware for autonomy is significantly higher than the price of a humanoid tall enough to hit the pedals in a normal car. Meanwhile, you can get more than 1 lidar and hopefully more accurate steering and acceleration control.

[matthias heyrman @MatthiasCaro]

@yuxuank02 Very cool! I'd love to see the pen spin on hardware if you have a video of it. Curious to see if this can also be expanded to transition between generated poses?