Runpei Dong

Intern @ Amazon FAR | CS PhD student @UofIllinois | Previously @Tsinghua_IIIS and XJTU | Interested in robot learning & machine learning

[Runpei Dong @RunpeiDong]

Want to ask your humanoid to get novel objects you want from a novel table? We introduce HERO, first achieving open-vocabulary visual loco-manipulation using human language queries! Now we can ask the humanoid to see and grasp the target object (eg, a carrot instead of a tissue box) with whole-body coordination.

How do we do it? Key designs and findings in this work:

We propose a residual-aware end-effector tracking policy that tracks the target accurately in a closed loop.
We find that the robot's forward kinematics are quite inaccurate, and thus we propose residual neural forward models that correct the end-effector FK and base leg odometry.
We design a modular system powered by visual foundation models that achieves generalizable grasping capabilities with 83% success rate on novel daily objects and daily scenes.

Check out our new project:
Learning Humanoid End-Effector Control for Open-Vocabulary Visual Loco-Manipulation
Project page: https://t.co/4QtbYne1I2
Paper: https://t.co/FLEjaGkRpW

[Hojune Kim @HojuneKim]

What if a phone scan is all you need to teach a humanoid a new skill, one that generalizes to scenes it's never seen?

Introducing 🦿LEGS🦿: a photorealistic loco-manipulation simulator. No teleop; policies deploy zero-shot on a Unitree G1 🤖

https://t.co/Ubfep7hP7l 👇

[Shuran Song @SongShuran]

I'm increasingly interested in the problem of *multisensory continual learning*, since it feels inevitable for robotics.

Unlike vision, many robot sensors (e.g., force/torque, tactile, audio) are highly task- and system- specific. It's unrealistic to expect a single pretraining dataset to contain every future sensor. And as robotics evolves, we'll keep building new sensors.

So the question is: Can we plug a new sensor into a pretrained vision-only foundation model without forgetting everything it already knows?

Better yet, can the new sensor actually improve the model's existing vision-based skills?

That's exactly the question that motivated MuSe 👇

[Zhongyu Li @ZhongyuLi4]

Why should collecting robot demonstrations require a robot at all? Introducing RynnWorld-Teleop -- a new paradigm we call Digital Teleoperation. Instead of teleoperating physical robots, operators control a robot-centric world model that generates synchronized robot videos and actions for imitation learning.

⚡ 40+ FPS real-time generation
🤖 Effective zero-shot Sim2Real
🌍 Hardware-agnostic, infinitely scalable robot data collection.

It is also a joint work with Haoyu Zhao & Damo Academy. Paper & code👉https://t.co/aimHEOcPWx

[Baoyu Li @BaoyuLi6]

Egocentric human data is abundant, but human motion is not always positive supervision for robot policy due to embodiment gaps. Naive BC co-training can HURT performance ☹️.

🌟Our key finding in **EgoWAM**: the state-prediction branch of a World Action Model effectively bridges this embodiment gap, enabling robot performance to scale with diverse **in-the-wild** human data.

💡The key question then becomes: what world representation transfers best across embodiments?

👇🏻Let’s take a deep dive into it:
🌐 https://t.co/VnhUs8CFKf

🧵[1/]

[Runpei Dong @RunpeiDong]

Very interesting results on in-context adaptation of humanoid robots!

[Nikhil Sobanbabu @NikhilSoban353]

🤖Humanoid robots, to be truly useful, must handle dynamics they never saw during training: payloads, slopes, soft terrain, and more..

But these are exactly the settings where the familiar sim-to-real pain also shows up, and every humanoid researcher has probably at least once said:
“but it worked perfectly in simulation!!”

For precise whole-body control, the challenge is not just avoiding failure. Small execution errors can quickly cascade into large deviations.

In FADA, we study how humanoids can adapt to these unseen dynamics. Our key observation is simple: the robot’s intent often transfers, but the execution does not.

1/N🧵

[Qingzhou Lu @Axell_wppr]

Humanoids should take on the heavy lifting jobs for humans. But can full-size humanoids handle heavy-payload teleoperation from noisy VR inputs?

Excited to introduce our work, HEFT: Heavy-Payload Full-size Humanoid Teleoperation.

HEFT tracks human intent from raw, noisy VR signals and enables real-world teleoperation with payloads up to 24 kg on L7, a 175 cm, 65 kg full-size humanoid.

Website & more demos: L7 heavy-payload teleop + G1/L7 high-dynamic tracking
https://t.co/fFgSWgpA7V

G1 & L7 training code/checkpoints:
https://t.co/uGimX29xyU

[Yana Wei ✈️ ICML 🇰🇷 @yanawei_]

🚀 Is MLLM perception evaluation really saturated? Check out our ICML 2026 paper!

While scores are getting harder to distinguish, models still make unacceptable visual mistakes in real-world use. Even one critical visual mistake can make the whole response unreliable to humans.

👉 We introduce PerceptionRubrics, a rubric-based evaluation framework for multimodal perception, which shows strong agreement with human Elo scores from VisionArena.

Our benchmark includes 1,038 information-dense images and 10K+ atomic rubrics across 7 domains: natural scenes, OCR, GUIs, charts, STEM, logic puzzles, and creative/cultural images.

We evaluate 20+ mainstream MLLMs, including GPT-5.5. Key findings:
1️⃣ Models often pass fragmented atomic checks but fail strict conjunctive perception, revealing a clear Reliability Gap.
2️⃣ Perceptual reliability remains especially challenging in information-dense domains such as GUIs, documents, and structured data.
3️⃣ As an automated benchmark, PerceptionRubrics shows strong alignment with human preference.

🏠 Project Page: https://t.co/kNjXaxf8Ym
📊 Code & Data: https://t.co/CjjrIUda7C
📄 Paper: https://t.co/sWf37vnz2C

Welcome model evaluations on PerceptionRubrics and would love to hear your feedback!

#ICML2026 #MLLM #MultimodalAI #ComputerVision

[Runpei Dong @RunpeiDong]

Incredibly robust and solid system 1 results with simple iPhone data, happy to see such a good generlaization in daily scenes. Congrats @wangyenjen!

[Yen-Jen Wang @wangyenjen]

How can we scale perception-based humanoid learning without collecting massive humanoid teleoperation data?

🚀 Excited to finally share VLK!

What excites me most about VLK is that it reframes data collection as a data generation problem. Instead of relying on expensive humanoid teleoperation, we automatically generate synchronized vision, language, and whole-body kinematics from reconstructed real-world scenes.

Making this vision a reality required bridging three fundamental challenges:

👀 Perception: Bridging the RGB sim → real gap through visual domain randomization and motion blur mitigation during both training and deployment.

🤖 Embodiment: Bridging the kinematics → dynamics gap with real-time VLA deployment, test-time RTC, and SceneBot, enabling seamless deployment on a real humanoid.

🌍 Environment: Bridging the real-world → synthetic gap to enable scalable Vision-Language-Kinematics data generation through scene reconstruction and interaction synthesis.

It has been an amazing journey working with such an incredible team. For a complete walkthrough of the project, check out @jiaman01's thread below 👇

🌐 Project: https://t.co/PnvpCDW4fi
📄 Paper: https://t.co/DPe20ilXm7
🎦 Video: https://t.co/BivXCxkzcq

Huge thanks to my amazing collaborators @jiaman01 @eric_srchen @TakaraTruong @ Pei Xu, and to our advisors @pabbeel @rocky_duan @KoushilSreenath @akanazawa @carlo_sferrazza @GuanyaShi @ckarenliu.

[Zhi (Leo) Wang @TX_Leo_Wang]

1/🧠Humans are the best robot data source — but video alone misses one thing: force.
2/🙁Tactile gloves capture force — but they're costly and block the real touch manipulation depends on.
3/💪Maybe the future of touch lives on your wrist: surface EMG reads the muscles that cause force — tactile sensing without ever touching a tactile sensor.
4/🔥Want a fully open-source framework — hardware + software — to train your own force-aware learn-from-human-data robot policy?

🚀We introduce ForceBand: Learning Forceful Manipulation with sEMG -- bring force into human videos with sEMG, for force-aware manipulation ⬇️

✦ Zero-Shot Human-to-Robot Transfer
✦ Force Beyond Vision
✦ Free-Hand Force Sensing
✦ Collect by Anyone, Anytime, Anywhere
✦ Deploy on Any Robot, Any Camera, Any Environment
✦ Open-Source & Low-Cost & Easy-to-Implement

Let's squeeze every bit of signal out of human data, and let robots feel the force!

🌐 Website: https://t.co/Mcs3fw9b4P
📄 Paper: https://t.co/QMCtkLgR2I
💻 Code: https://t.co/ecSTUC5ynn
🎥 Video: https://t.co/IPSaeqmmUE

🧵 1/n

[Xiaomeng Xu @XiaomengXu11]

🌱 How do you control a robot whose body is constantly growing, buckling, and reshaping?

📷 Put 19 cameras on it.

Meet PanoVine, the first autonomous vine robot system.

We distribute 19 cameras along a 6 meters, 7-DoF soft growing vine robot, giving it whole-body visual feedback of both itself and its surroundings. Using these observations, we train an end-to-end visuomotor policy that autonomously navigates branches, slopes, gaps, turns, and obstacles.

🔗 https://t.co/znQUj88VFu

[Danfei Xu @danfei_xu]

Check out our new work led by @Dantong_Niu: T-Rex = EgoScale + tactile sensing.

My view: tactile, force, and contact will be critical interfaces for human-to-robot transfer. We still lack good human tactile sensorization, but one step at a time.

[Runpei Dong @RunpeiDong]

Super cool results! Congrats!

[Siheng Zhao @SihengZhao]

🪜 What if humanoids could climb ladders and work on them straight out of simulation?

Meet LadderMan: a perceptive system for zero-shot sim-to-real ladder climbing and on-ladder manipulation.

Watch the humanoid climb, stabilize, and manipulate—all in one system. 🤖👇

[Runpei Dong @RunpeiDong]

Wonderful benchmark, congrats!@yining_hong

[Yining Hong @yining_hong]

Excited to share ESI-BENCH, a benchmark for Embodied Spatial Intelligence!

Most spatial reasoning benchmarks assume an oracle observer: the agent is given the right image, view, or 3D scene.
But in the real world, the observer is also an actor.
To understand space, agents must decide where to look, how to move, and when to interact, to reveal what is hidden: occlusions, containment, contact, dynamics, and functionality.
In many cases, the hard part is not perception itself, but choosing the right action to make informative perception possible.

ESI-BENCH tests this perception-action loop.
Agents receive an egocentric observation and a spatial question, then must actively gather evidence through perception, locomotion, and manipulationbefore answering.

The benchmark spans 10 task categories, 29 subcategories, and 3,081 instances, built in BEHAVIOR-1K across realistic interactive scenes.

🌍Webpage: https://t.co/Ou3zJ48eFx
💻Code & data: https://t.co/Mw0kU5hoyA

Thanks for collaborators: Jiageng, Han, @ManlingLi_ , Leonidas Guibas, @drfeifei , @jiajunwu_cs , @YejinChoinka

[Tairan He @TairanHe99]

GR00T-VisualSim2Real is now open source!

VIRAL and DoorMan are now available with training code, simulation assets, and the full recipe for bringing visual sim-to-real loco-manipulation skills to your own humanoids.

Repo: https://t.co/vgRsCeRG8w

[Peter Stone @PeterStone_TX]

I'm super excited and proud about this result. A fantastic team at Sony AI, led by Peter Duerr, put together just the right mix of science and engineering to create the first robot to beat a professional athlete in a real-world competitive sport!

[Yaru Niu @yaru_niu]

A touch-aware humanoid manipulation policy that cleans the lab for you🧹🧪

Introducing Humanoid Touch Dream: a real-world system for dexterous, contact-rich humanoid loco-manipulation.

Our key idea is simple: the policy predicts future hand forces and tactile latents alongside actions, within a single-stage training framework.

https://t.co/Pt5pXA65wm

1/7

[Shaowei Liu @stevenpg8]

📢MoRight: Motion Control Done Right

"What if your video model actually understood cause and effect?"
Existing motion-controlled video models entangle camera and object motion, and treat everything as kinematic displacement. MoRight changes both.

🔥 Motion Causality — MoRight decomposes motion into actions & consequences. Give an action → MoRight predicts consequences (aka motion simulation) . Give a desired outcome → MoRight recovers the driving action (aka motion planning). Not merely displacing pixels.

🎬 Disentangled Control — MoRight separates camera and object motion, allowing users to independently control each of them. No entanglement.

Project Page: https://t.co/IVIgJopCCI
Paper: https://t.co/t0kQqJfXQE

[Yining Hong @yining_hong]

I wrote a blog "Three Levels of TTT" — Test-Time Training, Meta Training, World Models, 3D & Self-Supervised Learning:
https://t.co/5eZQUKg2RH

The three levels are:
🧠 Episode — hippocampus encodes fast, neocortex consolidates slow. No labels needed.
🌱 Individual Lifetime — there is no train/test split. Every minute is testing as well as training.
🌍 Natural Selection & Evolution — continuous adaptation integrates into the species' prior.

Each level is the meta-training of the level below. Each level is the test-time training of the level above.
Priors flow down to the lower level; consolidated adaptations flow up to the next higher level.

🧠 The self-supervised signal needs no labels — it comes from the structure of experience itself: what did I expect vs. what happened? What follows what? What appears together?
🌱 This consolidates across a lifetime — every minute is testing as well as training. Given the priors of the human species, an infant develops 3D perception, object permanence, intuitive physics — not from instruction, but from reaching, crawling, acting. The world teaches the rest through self-supervised learning.
🌍 But what gives us those priors? Two front-facing eyes, exactly the right distance apart for depth to emerge. Pain and proprioception as free error signals. A face-detection circuit running at birth. Billions of years of test-time feedback from individual lives, accumulated and frozen into hardware. Evolution doesn't optimize behavior — it optimizes the prior you start from.

[Siyuan Huang @siyuanhuang95]

Excited to introduce OmniClone, a robust teleoperation system for humanoid mobile manipulation. While systems like TWIST2 and SONIC paved the way, we put efforts into solving the critical stability and scaling gaps.

1/ 📊 Moving past "vibe-based" testing. We’ve built a comprehensive diagnostic benchmark to systematically evaluate whole-body teleoperation. No more trial-and-error—get the actionable insights needed for true policy optimization.

2/ 👤 Universal Human-to-Robot Mapping. Teleop often breaks when switching operators. OmniClone mitigates biases from hardware fluctuations and, crucially, diverse human body shapes, ensuring high-stability control regardless of the person in the suit.

3/ 🚀 System Optimizations for Whole-body Manipulation Policy. By optimizing for affordability and reproducibility, OmniClone provides the high-fidelity pipeline necessary to collect data and train humanoid whole-body policies at scale.
fully
The model checkpoints and deploy code are now fully released—welcome to play with it! 📦

📄 Paper: https://t.co/kDm60WeuMD
🌐 Project: https://t.co/WGcfYridEs
💻 Code: https://t.co/U1QLgaipcd

[Shuran Song @SongShuran]

Turning a behavior "Prior" to a high-performing "Pro" in hours⚡️with DICE-RL (Distribution Contractive RL Finetuning)💡

Check out @s_zhanyi 's 🧵 for the secret source 😉