Jason Liu

PhD student @CMU_Robotics | Prev @NvidiaAI @UofT

[Jason Liu @JasonJZLiu]

💥Introducing FACTR 2, learning external force sensing on commodity robot arms without needing dedicated sensors.

We show that learned force signals enable force-feedback teleop on low-cost arms and improve BC policies.

FACTR 2 consists of:
1. Neural External Torque (NEXT): learns external forces without needing dedicated force sensors.
2. Force-Informed Re-Sampling Training (FIRST): uses the learned force signal to identify task-critical regions and upsample them during training.

w/ @StevenOh_ @_tonytao_

🧵(1/N)

[Jason Liu @JasonJZLiu]

Amazing contribution to the community, with really insightful findings as well

[Ritvik Singh @ritvik_singh9]

Introducing ABC: open data, training, and infrastructure for robotics.

We release the largest teleop dataset to date, and extensively investigate design decisions, pretraining, and post-training techniques.

@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall

[Changyi Lin @changyi_lin1]

Manipulation happens through surfaces. To understand contact-rich dexterous interaction, motion alone is not enough. We also need to know surface properties and contact state.

Excited to share ART-Glove, an articulated tactile glove that captures contact-grounded information while preserving human dexterity. It provides:

- Known Geometry: 16 rigid functional surfaces
- Surface Motion: 22 anatomically aligned joints
- Tactile Contact: 2048 piezoresistive taxels

Huge thanks to my advisor Ding @zhao__ding, and to Yuxiang @yxyang1995, Maria @bauzavillalonga, Marissa, and Peide @peide_huang for the valuable advice and discussions.

Paper: https://t.co/xSYCyqHrZf

[Wenli Xiao @_wenlixiao]

Autoresearch just left the sandbox and entered the embodied world.

We are excited to introduce 𝐄𝐍𝐏𝐈𝐑𝐄: a system that drops frontier coding agents onto a fleet of real robots and hands them the entire loop:

reset the environment → search the literature → implement ideas and build the infra → train and deploy → self-verify → analyze the logs and rewrite the code → repeat, until the policy is reliable in the real world. No human in the loop.

Guided only by the robot's self-proposed, heuristic-based success signal, the agents hill-climb to 99% on dexterous real-world tasks: organizing pins into a box, seating GPUs, tying zip-ties.

We envision the bottleneck in robotics shifting — from building smarter algorithms to building the closed physical feedback loops an agent can finally turn on its own.

🔗 https://t.co/3tL2ArGo3v
From @NVIDIA @CMU_Robotics @Berkeley_AI
🧵

[Litian Liang @litian_liang]

Introducing Universal Manipulation Exoskeleton (UME)

A low-cost exoskeleton with real-time haptic torque feedback for learning autonomous policies that perform highly force-mediated, tightly space-constrained, visually occluded, whole-body, and long-horizon mobile manipulation tasks.

Using UME, the teleoperator can unsheathe a heavy metal sword completely blindfolded.

https://t.co/W3PHmYRm4q

🧵1/N

[Jason Liu @JasonJZLiu]

We haven’t done these experiments, but from the perspective of training policies, we didn’t find much difference between using end effector wrench vs joint torque. But considering our joint torque predictions are accurate, I presume the transformed end effector wrenches will be as well.

[Jason Liu @JasonJZLiu]

💥Introducing FACTR 2, learning external force sensing on commodity robot arms without needing dedicated sensors.

We show that learned force signals enable force-feedback teleop on low-cost arms and improve BC policies.

FACTR 2 consists of:
1. Neural External Torque (NEXT): learns external forces without needing dedicated force sensors.
2. Force-Informed Re-Sampling Training (FIRST): uses the learned force signal to identify task-critical regions and upsample them during training.

w/ @StevenOh_ @_tonytao_

🧵(1/N)

[Jason Liu @JasonJZLiu]

@stevencheng You mean arms such as the SO-101?

[Jason Liu @JasonJZLiu]

@fnnBsch Do you mean evaluating the accuracy of the learned joint torques transformed to end effector forces? Or do you mean feeding end effector forces to the policy?

[Jason Liu @JasonJZLiu]

@raegher The piper arms cost ~$2.5k. The leader arms are ~$600

[Jason Liu @JasonJZLiu]

@VilleKuosmanen On the follower side, we don’t have these arms so we haven’t tried it ourselves, but I suspect it can work for these. On the leader side for force feedback, this is definitely possible.

[Jason Liu @JasonJZLiu]

Thanks! Hyperparameter tuning hasn’t been too bad since each NEXT model trains in ~1 min, so sweeps are cheap.

The training data should be free-space only, without contact. The model learns free-space dynamics, so contacts appear as residual external forces. Appendix A.3 has more data collection guidelines.

[Jason Liu @JasonJZLiu]

@signalgaining @ritvik_singh9 Most of the policy videos are with the AgileX Pipers

[Jason Liu @JasonJZLiu]

@VilleKuosmanen Feetech motors on the follower arms?

[Jason Liu @JasonJZLiu]

Here are a few more tasks we trained using the system

w/ @yangphiliphan

[Jason Liu @JasonJZLiu]

💥Introducing FACTR 2, learning external force sensing on commodity robot arms without needing dedicated sensors.

We show that learned force signals enable force-feedback teleop on low-cost arms and improve BC policies.

FACTR 2 consists of:
1. Neural External Torque (NEXT): learns external forces without needing dedicated force sensors.
2. Force-Informed Re-Sampling Training (FIRST): uses the learned force signal to identify task-critical regions and upsample them during training.

w/ @StevenOh_ @_tonytao_

🧵(1/N)

[Jason Liu @JasonJZLiu]

@squared2pi @StevenOh_ @_tonytao_ We ran ablations on the input signals and found that NEXT still works pretty well even without feeding in delta q, though performance is slightly better when it is included.

[Deepak Pathak @pathak2206]

Force is arguably the most overlooked ingredient in modern robot learning.

Introducing FACTR 2: it turns *any* commodity robot into a force-aware system with no force sensors required.

Train a tiny force network in <1min with <10mins of data and drop it into any existing teleop pipelines:

✅ Free force sensing for both the robot and the operator arm
✅ Makes demos higher-quality → fewer of them needed.
✅ A new force-aware learning algorithm (FIRST) uses those recovered forces to figure out which parts of a demo actually matter, making learning data-efficient.
✅ Strong performance on complex tasks with fewer demos and even no pretraining!

More details below.

[Kenny Shaw @kenny__shaw]

Manipulation policies should focus on contact!

FACTR 2 first learns force estimation for any robot arm without requiring any extra sensors.

It uses this to train BC policies that focus on the contact rich moments that matter most for success.

[Russ Salakhutdinov @rsalakhu]

New work on FACTR 2: Learning External Force Sensing for Commodity Robot Arms Improves Policy Learning:

Paper: https://t.co/7PXIUrXpAK
Web: https://t.co/IeJeTJ27L8

FACTR 2 shows that learned force signals can both enable force-feedback teleoperation on low-cost manipulators and improve behavior cloning (BC) policies for contact-rich tasks. It consists of two components:

1. Neural External Torque Estimation (NEXT): A lightweight model that infers external joint torques without dedicated force sensors.
2. Force-Informed Re-Sampling Training (FIRST): A training strategy that uses the learned force signal to identify and upsample task-critical moments.

The key insight is simple: policy failures rarely occur in free space, they occur during brief pre-contact alignment and contact-rich interactions, where precise corrections matter most.

Together, NEXT and FIRST bring force-aware teleoperation and robust long-horizon contact-rich policy learning to off-the-shelf robot arms, without requiring additional sensing hardware.

See a more detailed thread by @JasonJZLiu.

[Jason Liu @JasonJZLiu]

@lumbogg 😅

[Jason Liu @JasonJZLiu]

@CMU_Robotics @StevenOh_ @_tonytao_ @yangphiliphan @kenny__shaw @funabashihand @rsalakhu @pathak2206 Also check out @_tonytao_’s thread as well!

https://t.co/sYnSWe9v5h

[Tony Tao @_tonytao_]

What if some parts of a robot demonstration are more important than others?

Most of a trajectory is free-space motion. But success or failure is often determined by a few critical moments around contact.

In FACTR 2, we use force to find these moments and prioritize them for training. We find this helps policies learn better alignment and recovery behaviors, like the example below.

w/ @StevenOh_ @JasonJZLiu

🧵(1/N)

[Jason Liu @JasonJZLiu]

Special shoutout to @StevenOh_.

Steven has been visiting us at CMU from Japan for the past few months and worked incredibly hard on this project. I’m very proud of what he has accomplished, and excited for him to start his PhD at UChicago.

Please check out his thread for more details.

https://t.co/UZzgSIC9CZ

[Steven Oh @StevenOh_]

Force sensing for low-cost robot arms — without adding force sensors.

🚀 Excited to share FACTR 2! 🚀
FACTR 2 enables external torque sensing on low cost arms and uses it to improve policy learning.

w/ @JasonJZLiu  @_tonytao_
🧵(1/6)