Runhan Huang

PhD student @Harvard | Prev. @Tsinghua_IIIS (Yao Class) | Robot Learning, Generative AI

[Runhan Huang @RunhanH]

Honored to co-lead RAX with Xiangcheng (@xczhangxx)! Excited to share what robots can learn through scaling exploration—no demos, no model finetuning!

We’re open-sourcing the full framework and every experiment session. Please fell free to try out!

Project + sessions: https://t.co/C10K3FxlGV
Paper: https://t.co/MpZhZloU7H
GitHub: https://t.co/XGLeyZaz5v

Huge thanks to @du_yilun for your advice and support, and to all our amazing collaborators—it’s been a joy working with you!
@TonyC56348978 @yitongli165665 @wangyenjen @Lyubh22 @haotian_yeee @ChuyeHong @YXWangBot @Huangyu58589918 @Haoyu_Xiong_ @yangzhen04 @xieenze_jr @MandersonTravis @StefanoErmon @pabbeel Antonio Torralba @dawnsongtweets @Weirui_Ye

[Yilun Du @du_yilun]

Robotics has spent years scaling demonstration data and model size. What happens when we scale EXPLORATION for robot agents?

We're incredibly excited to share RAX, an agentic robotics system that learns by scaling online exploration.

A coding agent connects directly to the robot’s perception and control APIs, learns from feedback, and builds programs that perceive, plan, and act. The model stays frozen. As exploration scales, the agent improves its programs and builds up reusable skills.

The results:
• ZERO teleoperation or action demonstrations on real robots—from insertion with 0.1 mm clearance and air-fryer manipulation to deformable objects and long-horizon mobile whole-body control.
• +37 percentage points on average over leading learned policies across five simulation benchmarks: RoboDojo, BEHAVIOR-1K, HumanoidArena, DexJoCo’s bimanual tasks, and LIBERO-Pro.
• One framework spanning single-arm, bimanual, dexterous, humanoid, and mobile robots, with new embodiments connected through APIs.

The craziest part: on each real-world embodiment, going from scratch to mastering the tasks took roughly two hours. No teleop. No model finetuning.

We’re open-sourcing the full framework AND every experiment session. We can’t wait to see what people build with it!

Project + sessions: https://t.co/O04Z5PUgqY
Paper: https://t.co/sd66qnYZWB
GitHub: N

Project fully led by @RunhanH and @xczhangxx who are absolutely awesome.

How it works, experiments, and robot videos below 🧵

(1/N)

[Toru @ToruO_O]

How should we approach research, when the very field we work on is going through rapid and potentially fundamental changes?

I recently gave a talk at the Stanford Vision Seminar, reflecting on my PhD journey and how AI is changing the way I think about research. I adapted part of it into a new blog post, organized around three questions:
1. What should we work on?
2. How should we work with AI?
3. What is a PhD for?

I hope some of my reflections are helpful. The talk ended with an open discussion, and I’d love to continue it here; curious how others are thinking about these questions.

Personally -- as much as there's FOMO, I also think life is very long, and we have time.

https://t.co/DytwGOzBLy

[Sarah Liaw @liaw_sarah]

We introduce action learning: learning a stochastic action that scores how plausible an entire trajectory is, directly from observed trajectories.

No known forces. No potential energy function. No simulator likelihood.

We think this is a new paradigm for reasoning about trajectory data.

Joint work with @du_yilun and @YuanqiD.

🔥 Recovers underlying physical dynamics with over 2,000× lower error than local regression in a double well system.

🪐 Identifies a hidden 0.01% perturbation in the Sun's mass from trajectories of a 31 body orbital system.

🧬 Reuses a single learned molecular action for trajectory completion, steering, constrained optimization and temperature transfer.

The core idea: One learned action → reuse for many possible downstream tasks!

[Runhan Huang @RunhanH]

Yes, agents are more time-efficient in wall clock time than VLAs/WAMs too. Currently, the full exploration process in RAX takes around 2 hours per task, with most of that time spent on LLM reflection. While to fine-tuning a VLA or WAM model, it often requires 10+ hours of wall clock time GPU training, especially when VLA/WAM co-training on additional data to mitigate catastrophic forgetting.

[Runhan Huang @RunhanH]

🚀 RAX now supports DeepSeek Harness and Kimi Code as agent harnesses!

We’ve also added direct support for self-hosted DeepSeek-V4.1-Flash, so you can run RAX with local inference.

On the DexJoCo benchmark, RAX + DeepSeek-V4.1-Flash outperforms π0.5 and Fast-WAM by more than 10% with just 3 demonstrations. At API pricing, it costs roughly 1/20 as much as Claude Opus 5.5!

Try it out! 🤖

🌐 Project + sessions:
https://t.co/yS0mTf7T96
📄 Paper:
https://t.co/BSRB3PQjLb
💻 GitHub:
https://t.co/XGLeyZaz5v
🧩 Skill library:
https://t.co/tcbFIherKu
🤗 Dataset with full robot logs and sessions:
https://t.co/wXsHJ3TVfp

@deepseek_ai @Kimi_Moonshot

#Robotics #EmbodiedAI #AIAgents #OpenSource #DeepSeek #RobotLearning #LLM

[Yilun Du @du_yilun]

Robotics has spent years scaling demonstration data and model size. What happens when we scale EXPLORATION for robot agents?

We're incredibly excited to share RAX, an agentic robotics system that learns by scaling online exploration.

A coding agent connects directly to the robot’s perception and control APIs, learns from feedback, and builds programs that perceive, plan, and act. The model stays frozen. As exploration scales, the agent improves its programs and builds up reusable skills.

The results:
• ZERO teleoperation or action demonstrations on real robots—from insertion with 0.1 mm clearance and air-fryer manipulation to deformable objects and long-horizon mobile whole-body control.
• +37 percentage points on average over leading learned policies across five simulation benchmarks: RoboDojo, BEHAVIOR-1K, HumanoidArena, DexJoCo’s bimanual tasks, and LIBERO-Pro.
• One framework spanning single-arm, bimanual, dexterous, humanoid, and mobile robots, with new embodiments connected through APIs.

The craziest part: on each real-world embodiment, going from scratch to mastering the tasks took roughly two hours. No teleop. No model finetuning.

We’re open-sourcing the full framework AND every experiment session. We can’t wait to see what people build with it!

Project + sessions: https://t.co/O04Z5PUgqY
Paper: https://t.co/sd66qnYZWB
GitHub: N

Project fully led by @RunhanH and @xczhangxx who are absolutely awesome.

How it works, experiments, and robot videos below 🧵

(1/N)

[Runhan Huang @RunhanH]

@DibblaX Model providers’ own harnesses tend to be better tuned for their models, but generally speaking claude code is the best

[Runhan Huang @RunhanH]

@Sahilcore thx!

[Junhan Sun @DavidSunok47]

LeWM predicts action effects, but searches 9,000 sequences to act.

Introducing INTACT: an end-to-end unified JEPA that completes the other half of LeWM: intent to action.

Only 1 epoch. 0 search.
95.33% Direct macro SR.
2.9-5.5 ms inference.
~300× lower planning latency.

[Haoyu Xiong @ CoRL @Haoyu_Xiong_]

After seeing Astra's strong reasoning capabilities in robotics, I'm more convinced that we shall not bake "language following / reasoning" into low-level action models, i.e., VLAs.

[Runhan Huang @RunhanH]

@Benogju We use RB-Y1 from Rainbow Robotics, Franka arm as our real-world experiment hardware! For the Deepseek DexJoCo benchmark, the hardware is Franka + Allegro Hand.

[zeji yi @ZejiYi]

Common belief: Robot policies need continuous action heads; discrete actions are 𝐥𝐞𝐬𝐬 𝐧𝐚𝐭𝐢𝐯𝐞 𝐚𝐧𝐝 𝐥𝐞𝐬𝐬 𝐩𝐫𝐞𝐜𝐢𝐬𝐞.

❗We find policies w/ the simplest discrete representations 𝐦𝐚𝐭𝐜𝐡 𝐨𝐫 𝐛𝐞𝐚𝐭 continuous ones—in sim & real, in single-step & diffusion/flow settings.

Here's why 🧵

[Guowei Xu @Kevin_GuoweiXu]

🚀 Where does intelligence reside?
The paradigm in modern AI suggests: scale up a single model, train it on more data, and general intelligence may emerge. Yet Society of Mind proposed another possibility: intelligence may emerge from interactions of less intelligent agents.
As we keep scaling up a few generalist agents shared by everyone, can we scale up societies of decentralized agents that work together and collectively achieve more than any one of them could alone?
We introduce MPT: Market-based Post-Training, a framework to train such societies of agents by jointly learning agent policies and self-orchestration in a decentralized manner.

[Runhan Huang @RunhanH]

Congrats! Excited to see RAX already deployed and working well on a totally new embodiment and task, just within one day after our release!

Please try out if you are interested in agentic robotics!

Project + sessions: https://t.co/yS0mTf7T96
Paper: https://t.co/MpZhZloU7H
GitHub: https://t.co/XGLeyZaz5v
Skill Library: https://t.co/tcbFIherKu
Huggingface dataset with full robot logs and sessions! https://t.co/wXsHJ3TVfp

#RobotLearning #Robotics #AI #MachineLearning #LLM #Opensource #AIAgents

[Shaoting Zhu @ShaotingZ38103]

@du_yilun We’ve deployed this pipeline on our own robot, and it works incredibly well! I believe this represents a revolution in robotics.

[Runhan Huang @RunhanH]

@ManjuH9955 @du_yilun Agree

[Runhan Huang @RunhanH]

@mask1we @du_yilun Yes it is, we are studying the scaling law of embodied agents

[Zhenting Qi @ZhentingQi]

Research index (e.g. Google Scholar) is one of the largest collaborative systems where millions of human users build on each other's work across centuries.

How about we build something similar for async LLM agents?

We give agents a minimal hypertextual system. They asynchronously search for and read related work, cite the work that they find useful, and publish (mini-)papers.

Each search returns only the top k papers, ranked by relevance and a PageRank-like citation weight, which bounds the read cost of each agent to O(k), independent of agent count.

We show that performance effectively scales as we keep adding agents, and the system is robust to absence/failures of individual agents.

[Runhan Huang @RunhanH]

@ShaotingZ38103 @du_yilun Congrats! Excited to see that RAX can be deployed easily into a new embodiment and a new task!