rr.420

just floating like floating point errors

[Jiangran Lyu @JiangranLyu]

6/6 at CoRL 2026 🎉

Our works span multiple layers of Physical Intelligence — from World-Action Models, agentic world generation, and cross-embodiment VLAs, to humanoid control and dexterous manipulation:

• WAM-TTT — learning from unlabeled human videos via test-time training
• GIF — agentically generating interactive, functional worlds at scale
• Cross-Embodiment VLA — zero-shot generalization to unseen robot embodiments
• HumanoidNDM — neural dynamics models for closing the sim-to-real dynamics gap
• BiDexGrasp — coordinated bimanual dexterous grasping
• KPGrasp — scalable dexterous grasp generation with keypoints + flow matching
Huge thanks to all collaborators for carrying these projects forward 🙏
Onward to deeper and broader Physical Intelligence 🤖🌍
#CoRL2026 #EmbodiedAI #PhysicalAI #WorldModels #VLA #Robotics

[Yunfei Xie @xiynfi1520580]

🤖 Introducing RoboTok, the “TikTok for robots.”

Just as TikTok recommends videos to people, RoboTok recommends relevant human demonstrations for robot learning.

Robot learning needs broad and diverse demonstrations, but collecting robot data is expensive. RoboTok is an internet-scale data engine that uses web video as a scalable and continuously growing source of demonstrations for dexterous manipulation learning.

Given one human demonstration video as a query, RoboTok retrieves other web videos with similar underlying manipulation motions.

Rather than matching videos by labels or visual appearance, RoboTok compares how the hands move over time.

💡 The key idea is to represent each video with canonicalized 3D hand trajectories. Each trajectory is expressed in an estimated actor-centered reference frame, so the movement is described relative to the person rather than the camera. This makes similar manipulation behaviors easier to compare across different viewpoints and scenes, even when the actor is partly occluded.

In our experiments, RoboTok retrieved more manipulation-relevant demonstrations than existing robot-data retrieval methods. When those videos were used to guide robot training, the simulated robots completed manipulation tasks more successfully.

I’m sincerely grateful to Howard Qian and Kaiyu Hang for leading this project. I also want to thank Yiting Chen, Kejia Ren, Podshara Chanrungmaneekul, Gaotian Wang, Bowen Wen @bowenwen_me, and Chen Wei @_Chen_Wei_ for their guidance and collaboration.

Project site: https://t.co/v43CCf0040
Paper: https://t.co/WZGwQLhDTK
Code: https://t.co/MfYLTxHopH
Data and models: https://t.co/i0s7RcrXvX

[RoboPapers @RoboPapers]

Reinforcement learning has unlocked tremendous gains in broader AI and humanoid robot whole body control, but still has yet to deliver strong results for robot manipulation due to the difficulty of capturing tasks in an accurate simulation; one way to fix this is to create complex environments with learned world models. But learning fully within world models is often intractable as powerful world models are computationally too expensive.

Instead, @Jsphamigo and @Rk4342R propose to break the problem down: into a large-scale world model that can generate forward trajectories, and a lightweight, low-dimensional latent-space model which can approximate local dynamics of a problem, without needing to back-propagate through a heavy global model. What this means: you can do reinforcement learning in this coupled local-global world model, and learn contact rich skills for a robot.

To learn more, watch Episode #101 of RoboPapers, with @micoolcho and @DJiafei!

[Jay Chooi @chooi_jeq]

GPT-6 Astra scored 95% on a robot control task, up from Fable 5.1's 40%, with 6.2x fewer output tokens at 2.3x lower cost. 🧵

[Alec Helbling @alec_helbling]

A while back I made a blog about rectified flows.

It is quite relevant to this post.

https://t.co/Xo8seA0sbq

[Lou @louszbd]

We honestly didn't see blender demo would get so much attention. Actually it is for a test of overall model capabilites especially coding in long horizon task.

We started GLM-5.3-Flash in an empty folder and let it run for 12 hours without stepping in. By the end it built this blender scene, used around 100 million tokens.

GLM wrote python scripts through the blender cli and created .blend file. In the loop it kept checking rendered images and making changes, until the result looked good. No mcp was used for the whole run.

Before writing the prompt, we went through several rounds of discussion with GLM about how to shape the scene. I’ve talked in previous post that it's always better to fully ask and understand “what” before asking “how” to make GLM know you better(than you do). After discussing, we decided to use 16 fixed camera views, and something others that should go into prompt.

When we wrote the prompt, we tried to describe what a good result should look like in as much detail as possible. This seems more important than modeling instructions which we think GLM now can handle well inside.

The whole process might feel a little scary if you never tried it before. You can start by giving GLM our template and your own idea. Ask it help you with prompt.

We will include the full prompt for skyline bar demo at the end as reference. Hope you can enjoy and help improve. And please share more tips and thoughts if you have good ones:)

Prompt as reference: https://t.co/xlgS7e18az

[Kadir Yilmaz @KadirYilmaz_CV]

Could this be the ViT moment for 3D scene understanding? 🚀

We revisit the good old Transformer architecture and apply it to 3D scene understanding with minimal modifications. #Volt ⚡️

Project page: https://t.co/1HYyTrtdak
Arxiv link: https://t.co/imcuvM6j0X

(1/5)

[Hitesh @heetezition]

teach a robot a new behavior the way you’d teach another person: show it once.

robotics paper · in-context robot learning >

“Behavior Prompting Policy: Demonstrations as Prompts for Manipulation”

BPP takes a fresh human demonstration at inference time and uses it directly as context for the policy. no task-specific fine-tuning.

with iPhUMI, you physically demonstrate the behavior, hand that experience to the robot, and let it attempt the same thing.

this feels much closer to “watch me, now you try” than the usual collect → fine-tune → deploy loop.

https://t.co/gQcL1A7jjT

[Jesse Zhang @Jesse_Y_Zhang]

We’re trying something new at the R2RL Workshop at CoRL 2026. 🔗 https://t.co/VyuH7wbDoR

No senior-speakers, no passive audience.

Key Q: What are the real bottlenecks in Sample-efficient, Real-World RL?

🧑‍🔬 No senior speakers. Students & scientists who actually deploy RL on robots.
🗳️ Crowdsourced problems. Audience proposes challenges, we all vote on what to focus on.
🧩Breakout sessions. Audience helps to propose and discuss solutions for top-voted problems.
🎤 Panel. The breakout discussions determine what we debate.
📝 Whitepaper: a community-constructed whitepaper summarizing the most urgent challenges for Real-World RL in the next year.

Submit your papers!
Attend to help us figure out what we should actually be solving next!

Submission deadline: September 30

[LeRobot @LeRobotHF]

$28 3D printed teleoperator device. And of course it's open source!

The design features clean cable routing, same resting position as the SO-ARM101 and optional rubber bands that improve teleoperation experience (great idea).
Used with MuJoCo simulator in the video below but can also be used with the follower arm.

Awesome contribution from our community member!
Check out the project here: https://t.co/1srEOAZTd0

[csgm @csgbwk]

Parallel grippers are big! The current SO-101 grippers are the most limiting part of the build imo (apart from the the f**tech servos)

[York Yang @YorkYang5050]

Some recent thoughts.

ICL is useful. RL is useful. VLA, sim2real, world models, memory—all of these technologies can contribute to building general-purpose robots.

But we should remember that a general-purpose robot is still a robot: a physical product that ultimately needs to be deployed and deliver real value to customers.

Once you look at the actual challenges of deployment, it becomes clear that no single technique is enough. It requires a full system, with careful attention to detail across models, hardware, infrastructure, reliability, and operations. Even at the model level, it’s unlikely that any single idea will be the answer. VLA, RL, sim2real, world models, ICL, memory, and many other approaches will likely all play a role. The real challenge is combining them in the right way, so they complement each other and maximize the capability and utility of the overall system.

So IMO, there’s no need to overhype any single technical term. This industry would benefit from staying calm and patient, and focusing on building the holistic system that can actually work reliably in the real world and create real customer value.

[Deepak Pathak @deepakpathak]

One surprising aspect of S1’s in-context learning (ICL) is where it shines most: super long-horizon tasks (10+ min) and scenarios outside the pretraining distribution.

For short, simple pick-and-place tasks (5–20 sec), most frontier models can already perform well either zero-shot with no prompt or from just a language prompt. With enough pretraining data, the model often already “knows” what needs to be done. @DrJimFan also highlighted this distinction eloquently here: https://t.co/vbZWh3NmTy

Once you move to genuinely unseen scenarios and long-horizon tasks, in-context learning becomes critical. This shows up dramatically in the scaling laws: S1’s ICL advantage over regular VLAs grows exponentially as we move further out of distribution.

We discuss this in detail in the S1 blog:
https://t.co/xpUtioZM5K

[Jim Fan @DrJimFan]

Seeing a hype wave around GEN-1.5, and rightfully so. Lots of respect to Pete & Andy for executing so well. The secret is in the naturally repetitive motions in human-collected data. There're 2 main sources for such repetitions:

(1) Symmetric patterns. Sorting, tidying, and assembling almost never finish in one motion. Open any assembly manual from IKEA, and you find most objects symmetrical. You drive one bolt, then its twin, then the next pair. Every {bolt A, bolt B} pair is a natural continuation in context, and the second instance is a free training signal that imitates the first ("prompt").

(2) Recovery. Humans drop things all the time, but we pick them up so fast, we don’t even notice. That reflex to fix is half of our physical competence. The key insight is to keep the failed first half instead of trimming it away. If the model consumes the full arc, fumble, catch, continue, then recovery shows up organically at test time. It's funny that in-context improvement results from *NOT* over-sanitizing your data.

The other critical ingredient is UMI. I've been saying for a while that teleop will not last, and GEN-1.5 is driving the final nail in the coffin. UMI is essentially a human wearing the robot gripper to collect data directly (human → data). Teleop inserts a layer of separation: human → VR/skeletal device → robot → data, which bleeds out all the human "physical intuition". The subtle sleight of hand we perform constantly with objects, the micro-adjustments, the feel of a part snapping into place, is nearly impossible to capture when you can't feel the environment directly.

Once you have enough data, many behaviors can actually be zero-shot. For example, you don't even need finetuning to pick up a novel object. The model "just knows" what to do given a similar scene in the training distribution. Whether in-context learning truly works or not also depends on how far away the test is from training. Currently, the demos are still a bit too simple to conclude.

I'm cautiously optimistic. Still, it's a great day in robotics.

[Hokin Deng @DengHokin]

Accepted at ECCV 2026: "Demystifying Video Reasoning"

On Friday Aug 28, first author Ruisi Wang @Rui147000038622 the work at our journal club. The finding: video models don't reason across frames (Chain-of-Frames) — they reason along diffusion denoising steps (Chain-of-Steps).

Paper: https://t.co/D4e3lhkI35

Fri Aug 28 · 7:30 PM PT
RSVP: https://t.co/DSQrJAHFov

[Junwei Liang 梁俊卫 @JunweiLiangCMU]

Can robots perform an unseen task using in-context learning, like LLMs do?

We introduce Zero-WAM, an in-context learning paradigm for robot manipulation: a human video serves as the prompt, showing what to manipulate, how, and in what order.

The key is HumanGen, a scalable pipeline that converts task-sampled robot trajectories into semantically matched human demonstrations—74.2K human–robot pairs across 8.6K tasks.

Zero-WAM reaches 46.95% zero-shot success on 7 unseen tasks (+29.5 pts from Lingbot-VA).

📄 Paper: https://t.co/nUZZGd4Znz
🌐 Project: https://t.co/LgPuZQvZa2
💻 GitHub: https://t.co/3GHcHzQxtX
#RobotLearning #EmbodiedAI #WorldModels #InContextLearning #Robotics

[rr.420 @tweetbyMLMan]

good

[Lerrel Pinto @LerrelPinto]

Turns out that doing In-context learning for robots is not that hard...

[Pollen Robotics @pollenrobotics]

Train in sim. Run it for real. Teach it new tricks. 🦆

This is the sim2real loop that powers Microduck.

RL stack open source here : https://t.co/mveM7y5fYa

Buy it here : https://t.co/Rmk8F4vpKd

Git : https://t.co/kcoCKdBi52

Join our community : https://t.co/TcX5uaf0yL

[Ryohei Sasaki@engineer @rsasaki0109]

ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video(ECCV2026)

ReViV reconstructs viewer-centric human motion (body, hand, and gaze) and view-centric scene geometry (camera and depth) from a single egocentric RGB video in a unified feed-forward model. For visualization, the initial camera pose and camera intrinsics are estimated with GeoCalib.

[Lightwheel @LightwheelAI]

Most egocentric data at scale is private. We think that's worth changing.

With @huggingface, we've released the first 10,000 hours of EgoSuite-Open100K — an open egocentric human dataset for Physical AI.

https://t.co/WSX5uMVIUg

#Robotics