EmPRISE Lab

EmPRISE Lab in CS at Cornell University. We are a full-stack robotics lab. Our vision is to EMpower People with Robots and Intelligent Shared Experiences.

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

Excited to announce that registration is open for the PhyRC 2027 Challenge, our physical robotic caregiving competition, now co-organized by the @EmpriseLab at @Cornell_Bowers and the RIRO Lab at KAIST.

One track: mobile manipulation for robot-assisted dressing. A seated manikin, a closed-topology T-shirt, and a robot that has to thread it over the body and dress both arms.

Phase 1 runs in NVIDIA Isaac Sim through 15 Dec. Top teams move to real robots in 2027.

We thank @hellorobotinc for generously sponsoring a Stretch 4 robot for the winning team!

56 teams from 17 countries entered the inaugural edition in 2025, which finished with live finals at ICRA 2025.

Registration closes 15 Oct: https://t.co/k9d8v1XUAk

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

What happens when a robot becomes so autonomous that it effectively removes the human from their own care?

Excited to share E-MPC, new work from @jiaying_fang0 and team at @EmpriseLab in #RSS2026. E-MPC plans not only for task success, but also for how much involvement an individual user wants and how much workload they are willing to tolerate.

One of the most thought-provoking aspects of the work is that the robot distinguishes between questions that are necessary for completing the task and low-burden questions intended primarily to preserve engagement. In other words, not every interaction needs to make the robot more capable. Some interactions exist to prevent the human from becoming a passive observer.

E-MPC treats engagement as something to regulate toward a personalized target, not simply maximize. In a real-robot bite-acquisition study, E-MPC significantly improved participants’ self-reported interaction satisfaction without reducing task-success ratings, and 9 out of 10 participants preferred it over the baseline.

This work asks an important question for the future of physical robot caregiving:

A robot may complete the task successfully, but did the person remain meaningfully involved?

Congratulations to the entire team @EmpriseLab!

@Cornell_CS @Cornell_Bowers

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

One of the most important lessons in physical robot caregiving is that more help is not always better.

I am proud to share our #RSS2026 paper, “Knowing When Not to Help: Active Estimation of Human Reachability for Just-Right Robot Assistance,” led by @yy2244_cornell and @RealZiangLiu. What excites me most about this work is its reframing of physical robot assistance as a problem of calibration, not simply intervention. A physical caregiving robot must learn not only when and how to help, but also what an individual can do independently and when it should step back.

The paper develops a personalized model of human reachability by combining biomechanical simulation, clinical knowledge from rehabilitation literature, and Bayesian active learning. Rather than requiring motion capture or an extensive offline calibration session, the robot actively selects informative configurations to learn a user’s reachability from online interactions.

In our real-robot studies, this “just-right” assistance significantly improved participants’ perceived physical engagement while maintaining task success and moderate workload.

To me, this represents an important direction for physical robot caregiving: systems that do not simply optimize task completion, but also preserve participation, autonomy, and human agency. Congratulations to @yy2244_cornell, @RealZiangLiu and the entire team @EmpriseLab, and many thanks to our collaborators at Binghamton University and Columbia University Irving Medical Center!

@Cornell_CS @Cornell_Bowers

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

Check out this work from @ZhanxinWu0725 and the team @EmpriseLab presented at #RSS2026.

E²CARE addresses a central challenge in physical caregiving robotics: how to transfer skills across tasks, environments, and robot embodiments while preserving context-sensitive safety.

The key idea is to represent primitive skills as reusable interaction templates that are adapted online using a unified 3D scene representation, language-model-generated constraints, and control barrier functions. This enables behaviors that are not only collision-aware, but also socially and functionally appropriate. For example, backing away when another person approaches during feeding or adjusting motion to preserve the user’s line of sight.

Evaluated across more than 100 simulated household environments and real-world studies with Franka and Kinova Gen 3 robots, the framework is a promising step toward adaptable physical caregiving systems.

Congratulations to @ZhanxinWu0725 and the entire team!

@Cornell_CS @Cornell_Bowers

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

Check out the RAG-Diff work by @Nekovowo and the team at #RSS2026.

What I find most interesting about RAG-Diff is its treatment of user preferences as two distinct forms of information. Some are explicit (language requests, force limits, regions to avoid, or spatial goals). Others are implicit in prior behavior: what did this person prefer in a similar situation?

RAG-Diff handles both at test time. PrefMem retrieves a relevant state–action snippet and its constraint annotation. I-Atten injects the snippet into the frozen diffusion policy’s cross-attention, while gradient-based value guidance steers denoising toward satisfying the explicit constraint.

This combination enables a frozen policy to adapt online across interaction, affordance, spatial, and semantic preferences.

@EmpriseLab @Cornell_CS @Cornell_Bowers

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

Proud of @madrish02 and the entire team's work on TACTIC. What I find most compelling about TACTIC is that contact is not treated as just another sensor input or a penalty term, it is built into every layer of the planning stack.

In the representation, RGB-D, proprioception, and distributed tactile sensing are fused through a proximity mask that focuses the model on the interaction geometry that actually matters: where contact is happening, and where it may happen next.

In sampling, contact Jacobians shape MPPI exploration, biasing candidate actions toward directions that can meaningfully regulate forces and steering unsafe samples away from high-force contacts.

In prediction, hybrid rollouts combine learned interaction dynamics with analytical robot kinematics: learn what is hard to model, while preserving the structure we already know. Interestingly, this hybrid variant idea that adds a kinematics-based goal cost during planning also improves planning with other learned world models, including V-JEPA2 on Reach and DINO-WM on Granular.

And in scoring, trajectories are evaluated not only for task progress, but also for contact evolution, force regulation, joint limits, and collisions.

That contact-centricity across representation, sampling, prediction, and scoring is, to me, the most exciting part of TACTIC. Very proud of the team and excited to see this work at #RSS2026! @EmpriseLab @Cornell_Bowers @Cornell_CS

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

EmPRISE Lab will have a strong presence at #RSS2026, with 5 main-conference papers plus workshop papers, talks, and panels. Here’s where to find our main conference work:

Monday, July 13: Session "World Models and Memory"

Check out RAG-Diff, led by @Nekovowo. RAG-Diff is a retrieval-augmented framework for adapting a frozen diffusion policy at test time to dynamic, personalized user preferences in caregiving-like environments. It retrieves relevant examples and constraints from a memory bank (PrefMem), using in-place attention re-computation (I-Atten) for implicit preferences and value guidance for explicit preferences.

Website: https://t.co/zNYCkBU0MP

Tuesday, July 14: Session "Manipulation 2"

Check out TACTIC, led by @madrish02. TACTIC is a receding-horizon controller for whole-arm manipulation. TACTIC combines vision and distributed tactile sensing in a contact-centric representation, then uses contact-aware action sampling and a hybrid latent/analytical predictive model to balance task progress with whole-arm force regulation.

Website: https://t.co/aDrsOMJASV

Wednesday, July 15: Session "HRI"

@ZhanxinWu0725 will present E2Care, a context-aware physical caregiving system designed to operate safely across changing tasks, environments, and robot embodiments. It treats caregiving primitives as interaction templates that can be reshaped online according to context.

Website: https://t.co/tU6K2i3lpk

@jiaying_fang0 will present E-MPC: An Engagement-Aware Human-in-the-Loop Framework for Robotic Systems. E-MPC recognizes that task success alone is insufficient for human-centered robots: users may still feel passive, disengaged, or overloaded. It jointly reasons about task success, engagement, and workload to decide when and how to involve the user.

Website: https://t.co/peHetmjlMW

Finally, check out “Knowing When Not to Help," led by @RealZiangLiu and Yunting Yan, exploring a central challenge in robot-assisted caregiving: providing help only when it is needed. Its Bayesian active-learning framework combines biomechanical models, clinical knowledge, and LLM-guided priors to learn user-specific reachability online and support “just-right” assistance while maintaining reliable task performance.

Website: https://t.co/pBgbZFALBx

You can also find us at the workshops:
• Rethinking What It Means to Be Safe for Generalist Robots
• Beyond the Lab: Rethinking Methods for Human Behavior Monitoring and Modeling in In-the-Wild HRI
• Towards Robust Execution of Long-Horizon Whole-Body Control Tasks
• Planning and Control with Imperfect Sensors and Perception

Due to visa complications, Ruolin, Rishabh, and Yunting will not be attending in person. Zhanxin, Jiaying, Joyce Yang, and I will be there, and will also present on their behalf.

Come say hi! :-)

@EmpriseLab @Cornell_CS @Cornell_Bowers

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

Physical caregiving is one of robotics' hardest frontiers: it is contact-rich, physically intensive, long-horizon, safety-critical, and full of deformable objects.

Physical caregiving tasks such as bathing, dressing, transferring, toileting, and grooming require professional training and considerable practical experience. Yet, no existing dataset captures how expert caregivers perceive, interact, and adapt in real-time when performing these tasks, in a form that robots can learn from.

✨ We introduce OpenRoboCare at #IROS2025, the first expert-collected, multi-task, multimodal dataset for physical robot caregiving, featuring:
🩺 21 expert occupational therapists demonstrating caregiving procedures
🛠️ 15 caregiving tasks across 5 Activities of Daily Living (bathing, dressing, transferring, toileting, grooming)
🧍 2 hospital-grade manikins for safety and repeatability
🎥 5 synchronized sensing modalities: RGB-D, pose tracking, eye gaze, tactile sensing, and expert task & action annotations
📂 315 sessions · 19.8 hrs · 31,185 samples

Beyond raw data, OpenRoboCare distills core physical caregiving insights:
- 3 core principles followed by occupational therapists: pre-positioning, anticipation of body mechanics, and task efficiency.
- 4 key physical techniques: the bridge strategy, segmental rolling, wheelchair recline, and stabilization of key control points.
- Quantitative patterns in task duration, predictive gaze behavior that precedes physical contact, and the timing, magnitude, and spatial distribution of contact forces across body regions and task phases.

The dataset will be made openly accessible through the AWS Open Data Sponsorship Program soon.

🌐 Check out our project website for more visuals and insights: https://t.co/MS3YNEolzQ

This work is led by: @xiaoyul14, @RealZiangLiu, and Kelvin Lin. This is a collaboration with Harold Soh's group from NUS and @DimitropoulouDr from CUIMC.

@EmpriseLab @Cornell_CS @IROS2025 @awscloud

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

Robots often rely on whole-arm contact for caregiving tasks such as bed-bathing and transferring. Humans exhibit contact preferences in terms of where and how much force to exert for comfortable interactions. But comfort isn’t one-size-fits-all: preferences vary across people, and for the same person, across different body parts. How can a robot adapt to these preferences on the fly?

Excited to share our work “PrioriTouch: Adapting to User Contact Preferences for Whole-Arm Physical Human-Robot Interaction” led by @rishabhmadan96! #CoRL2025

💡 Core idea
Treat user contact preferences as a ranking over control objectives. PrioriTouch learns this priority ordering online and executes it with Hierarchical Operational Space Control (H-OSC), so higher-priority contacts (the ones closer to causing discomfort) are protected while others yield.

🧠 Learning to rank, safely
We introduce LinUCB-Rank, a contextual bandit that updates the priority ordering from sparse user feedback (“I feel uncomfortable around my abdomen”).

To keep people safe, risky exploration occurs first in a digital twin (simulation-in-the-loop) and then is deployed on the real robot.

📊 What does this buy us?
A sample-efficient way of reasoning about contact preferences. In sim and hardware: fewer force-threshold violations, fewer feedback signals to reach the right ordering, and sustained task efficiency.

@EmpriseLab @ToyotaResearch @Cornell_CS @corl_conf

🗣️ Spotlight: Sep 29 (Session 4)
📊 Poster: Sep 29 (Session 2)

🌐 Website: https://t.co/DKazdptqbR
📝 Paper: https://t.co/a4MRp6hcZp

(1/3) 🧵

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

Introducing CLAMP: : a device, dataset, and model that bring large-scale, in-the-wild multimodal haptics to real robots. Haptics / Tactile data is more than just force or surface texture, and capturing this multimodal haptic information can be useful for robot manipulation.

Check out @pranavnnt’s work “CLAMP: Crowdsourcing a LArge-scale in-the-wild haptic dataset with an open-source device for Multimodal robot Perception”, at #CoRL2025.

The CLAMP device is an open-source, low-cost (<$200), portable (0.59 kg) tool that can sense 5 haptic modalities along with vision and language. Users can take it home and log haptic data via a PiTFT screen and buttons.

As far as we know, the CLAMP dataset is the largest multimodal haptic dataset in the robotics literature, with a total of 12.3 million data points from 5357 objects in 41 homes, collected by 16 CLAMP devices.

The CLAMP model is a material recognition model that outperformed GPT-4o, CLIP, and PG-VLM in our experiments, and generalized to haptic data from three different robot embodiments (WidowX and Franka with different grippers).

A finetuned CLAMP model enabled a 7-DoF Franka Panda to robustly perform three real-world manipulation tasks involving clutter, occlusion, and visual ambiguity.

@EmpriseLab @Cornell_CS @corl_conf

🗣️ Spotlight presentation at #CoRL2025 on Sep 30 (spotlight session 5)
📊 Poster session at #CoRL2025 on Sep 30 (poster session 3)
🌐 Website: https://t.co/T4v5JyBiz2
📄 Paper: https://t.co/9OFGgxg3ks

Check this thread for more details (1/6) 🧵

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

During a meal, food may cool down resulting in a change in its physical properties, even though visually it may look the same! How can robots reliably pick up food when it looks the same but feels different — such as steak 🥩getting firmer as it cools? 🍴

Check out @ZhanxinWu0725's work SAVOR: Skill Affordance Learning from Visuo-Haptic Perception for Robot-Assisted Bite Acquisition — an oral at #CoRL2025.

SAVOR introduces a novel method to learn skill affordances, which capture how suitable a manipulation skill (e.g., skewering, scooping) is for a utensil–food interaction. Skill affordances arise from the combination of tool affordances (what a utensil can do) and food affordances (what the food allows). Using this method, SAVOR improves bite acquisition success by 13% over state-of-the-art methods.

@EmpriseLab @Cornell_CS @corl_conf

🗣️ Oral presentation at #CoRL2025 — join us on Sep 28 (afternoon session)
🌐 Website: https://t.co/92Pwa5fWMy
📄 Paper: https://t.co/kFaGiNDamw

Check this thread for more details (1/6) 🧵

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

NERC 2025 is happening @Cornell this year. Here is the website with more details: https://t.co/tOClpYYz2G

We have a fantastic set of keynote speakers from a variety of backgrounds @Majumdar_Ani from @Princeton, Victoria Webster-Wood from @CarnegieMellon, @wendyju from @cornell_tech, and @HerlantLaura from RAI

With poster presentations of extended abstracts, Rising Star spotlight talks, and demos / booths and other support from our generous sponsors @FourierRobots @rai_inst @clearpathrobots @UnitreeRobotics @CornellCIS and @CornellCOE , this event is going to be exciting!

Do register (Early Deadline: September 10th, Late Deadline: October 3rd) and come enjoy this event at the beautiful @Cornell Campus in Ithaca on October 11th 🎉🎉

[Rajat Kumar Jenamani @rkjenamani]

Really excited to share that FEAST won the Best Paper Award at #RSS2025!

Huge thanks to everyone who’s shaped this work, from roboticists to care recipients, caregivers, and occupational therapists. ❤️

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

And we won the #RSS 2025 Best Paper Award! Congrats @rkjenamani and the entire @EmpriseLab team @CornellCIS 🎉🎉

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

Congrats @rkjenamani and the entire team @EmpriseLab on this impressive accomplishment and being nominated for Best Paper Award and Best Systems Paper Award at #RSS 2025!

This project took almost 2.5 years to get to this stage, and I am incredibly proud of what we have achieved in term of real-world deployment of a meal-assistance system with real users in their homes with minimal researcher intervention leveraging in-the-wild personalization.

Key insight is that for in-the-wild deployment of user-centered systems, adaptation and personalization need to go hand-in-hand with transparency and safety. More technical details are in the thread 🧵below.

@rkjenamani will be presenting this work on Monday (June 23, 2025) @RoboticsSciSys 2025 in the #HRI session. Do attend :-)

Website: https://t.co/YEgvvDhfwj
@CornellCIS @Cornell_CS @EmpriseLab

[Rajat Kumar Jenamani @rkjenamani]

Most assistive robots live in labs.
We want to change that.

FEAST enables care recipients to personalize mealtime assistance in-the-wild, with minimal researcher intervention across diverse in-home scenarios.

🏆 Outstanding Paper &amp; Systems Paper Finalist @RoboticsSciSys
🧵1/8

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

So proud of what we achieved with the PhyRC Challenge at #ICRA2025 with the support of our generous sponsors @KinovaRobotics and @hellorobotinc! From simulation to real-world assistive care tasks -- robot-assisted dressing and bed-bathing -- this was an ambitious undertaking. Huge congrats to all the teams who won a Stretch 3 robot (sponsored by @hellorobotinc) and a Gen 3 robot (sponsored by @KinovaRobotics) as prizes 👏. The teams were impressive but these tasks are extremely challenging, and no team was able to completely finish the tasks. Want to give these tasks a try during the next iteration of #PhyRC? Stay tuned for the next version :-) @EmpriseLab @CornellCIS

[Rajat Kumar Jenamani @rkjenamani]

Excited to share our work on continual, flexible, active, and safe robot personalization w/ @tomssilver, @RealZiangLiu, Ben Dodson &amp; @TapoBhat.

Also: @tomssilver is starting a lab at Princeton!! I HIGHLY recommend joining — thoughtful, kind, and an absolute joy to work with!

[Tom Silver @tomssilver]

Happy to share a new preprint: "Coloring Between the Lines: Personalization in the Null Space of Planning Constraints" w/ @rkjenamani, @RealZiangLiu, Ben Dodson, and @TapoBhat.

TLDR: We propose a method for continual, flexible, active, and safe robot personalization.

Links 👇

[Tapomayukh "Tapo" Bhattacharjee @TapoBhat]

Congrats @rohanbbanerjee and the team on getting nominated for Best paper award at #ICRA2025 for their work on “To ask or not to ask: Human-in-the-loop contextual bandits with applications in robot-assisted feeding”. Check out his presentation in Room 302 on Tuesday at the Awards Finalists 2 session at 11:15am.

In robot-assisted feeding (and many other real-world tasks in unstructured settings), full robot autonomy can be fragile as algorithms and systems may fail. Can a robot leverage a user, who is already present, for help vs. acting autonomously?

In our new work, we introduce LinUCB-QG: A human-in-the-loop contextual bandit algorithm that adapts querying based on both task uncertainty and a predicted user-specific workload based on a querying model learned using a newly collected diverse dataset.

Key takeaway is that users with mobility limitations may experience higher workload when queried compared to users without mobility limitations, so the robot queries less, prioritizing lower querying workload even at some cost to task performance. However, user satisfaction is high. Our system improves bite acquisition success while reducing user workload — validated via simulation and real-world user studies. This work was also done with @rkjenamani, Sid Vasudev, Amal Nanavati, @DimitropoulouDr, and Sarah Dean.

Website: https://t.co/AhcPTcvKt0

@EmpriseLab @CornellCIS @Cornell_CS @ieee_ras_icra @ieeeras