I am honored and grateful to receive an 𝗡𝗦𝗙 𝗖𝗔𝗥𝗘𝗘𝗥 award “𝘚𝘩𝘢𝘱𝘪𝘯𝘨 𝘌𝘮𝘣𝘰𝘥𝘪𝘦𝘥 𝘐𝘯𝘵𝘦𝘭𝘭𝘪𝘨𝘦𝘯𝘤𝘦 𝘛𝘩𝘳𝘰𝘶𝘨𝘩 𝘓𝘢𝘯𝘨𝘶𝘢𝘨𝘦-𝘎𝘶𝘪𝘥𝘦𝘥 𝘐𝘯𝘵𝘳𝘰𝘴𝘱𝘦𝘤𝘵𝘪𝘰𝘯.” 🎉
This project will support our work in the 𝗣𝗟𝗔𝗡 𝗟𝗮𝗯 toward more reliable, self-improving embodied AI systems. I am deeply thankful to @NSF, my students, whose creativity and hard work inspire me, as well as my collaborators, mentors, and @iSchoolUI colleagues for their support. I am also grateful to my former academic home @VT_CS where this work began to take shape. Excited for the work ahead!
Learn more about our lab 👉 https://t.co/xAFAe248Op
#NSFCAREER #NSFfunded #EmbodiedAI #MultimodalAI #ResponsibleAI #iSchoolUI #PLANLab
#iSchoolUI Assistant Professor Ismini Lourentzou has received a $600K @NSF CAREER award for her project, “Shaping Embodied Intelligence Through Language-Guided Introspection,” which aims to build safer and more reliable AI systems. 🤖
Congrats, Ismini! ▶️ https://t.co/6XHyuSbwFt
NEW RESEARCH: Predicting tuch is actually more useful than merely reading it!
This project involves @Ismini_L and others
Called DexTacWAM, for A Visuo-Tactile World-Action Model for Dexterous Manipulation, it is a method from UC Berkeley, UIUC, and Northwestern.
It puts fingertip touch inside a video world model rather than beside it.
Each fingertip's grayscale tactile map is painted as pseudo-RGB and pushed through a frozen pretrained VAE, the same encoder used for the camera streams and shared across all ten fingers.
A compressor with finger-identity embeddings, hand-pose conditioning, self-attention pooling and temporal refinement folds five fingertip streams per hand into a single latent, cutting DiT tokens from 3,168 to 864.
Those tactile latents are appended as extra views to a 2B-parameter pretrained video diffusion transformer, which is then trained with a modality-split flow-matching objective to denoise both the visual and the tactile future.
An action expert cross-attends to the world model's latents with per-modality K/V RMS normalization.
Training runs in three stages:
1. adapt the tactile encoder on a 488-episode, roughly four-hour bimanual corpus called Diverse-488 with the VAE frozen
2. finetune the world model on about 100 demonstrations per task
3. then train a fresh action expert on the same demonstrations.
The platform is a 22-DoF bimanual rig with Sharpa hands and vision-based fingertip tactile sensors, head and wrist cameras.
I find it super interesting that feeding the model touch does not bring much value, while making it predict touch actually enhances its performance.
VTAM uses an extra training target to encourage the robot policy to use touch.
It summarizes image motion and expansion. Contact cues, not force readings in newtons.
@Jensen_Yuan and team's Nov 5 revision. Synthetic example below.
https://t.co/utafglMHuR
Proud of @Jensen_Yuan for leading this work! 👏
🤖VTAM is our visuo-tactile world-action model for contact-rich manipulation, trained with ~100 demos per task and no tactile pretraining.
Code and data are open-sourced👇
Can a world-action model learn to predict touch without any tactile pretraining?
We introduce VTAM, an efficient visuo-tactile world-action model. With ~100 demos per task and no tactile or action pretraining, it is comparable to tactile models pretrained on 30K+ hours of visuo-tactile robot data, and is 6× faster at inference. 🧵
📄Paper: https://t.co/99DkVVGBJD
🌐Web: https://t.co/6drcq3y0oI
Code and data are open source!
💻 Code: https://t.co/McHqMi4DAc
🤗 Data (xArm + GelSight, 3 tasks): https://t.co/kuPR4BeQQ8
Can a world-action model learn to predict touch without any tactile pretraining?
We introduce VTAM, an efficient visuo-tactile world-action model. With ~100 demos per task and no tactile or action pretraining, it is comparable to tactile models pretrained on 30K+ hours of visuo-tactile robot data, and is 6× faster at inference. 🧵
📄Paper: https://t.co/99DkVVGBJD
🌐Web: https://t.co/6drcq3y0oI
Code and data are open source!
💻 Code: https://t.co/McHqMi4DAc
🤗 Data (xArm + GelSight, 3 tasks): https://t.co/kuPR4BeQQ8
Astra weight edit feels like another form of Richard Sutton-style continual learning: different parameters should adapt at different rates during deployment.
Here, most parameters effectively have a step size of 0, while only a tiny subset is updated. Sparse weight editing may be a practical path toward continual adaptation without full-model fine-tuning in real-world deployment.
📣 Working on robust AI, multitask learning, or multi-objective optimization? Bring your research to our #AAAI27 workshop!
🎤Check our fantastic speaker line!
📝 4 or 8 pages · Due Nov 20 (AoE)
📍 @RealAAAI Montréal + online
https://t.co/irIrVJyPe3
I will be sharing an overview of our research on 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐖𝐨𝐫𝐥𝐝 𝐌𝐨𝐝𝐞𝐥𝐬 at the @iSchoolUI Research Showcase tomorrow, Oct. 30.
📍Illini Union, Rooms A & B
⌚2:15–3:15 p.m. CT
Come learn about world models and robotics!
Join us for the #iSchoolUI Research Showcase on Oct. 30, from 11:30 a.m.–4:45 p.m. at the Illini Union! Explore human-centered, interdisciplinary research through keynote talks, short presentations, and posters. All are welcome!
Full schedule ▶️ https://t.co/pNcoSAUYWR
Both Sydney & Atlanta waitlists were activated on Oct. 27 at 10 am PDT. More registrations will be released on Oct. 30 10am PDT. Authors with unused reserved registration spots are reminded to register urgently.
💬 AI assistants need to reason while the conversation keeps changing.
Our new survey maps open problems and new research directions for Native Real-Time Interactive Omni Intelligence.
🌐https://t.co/sXhy9w7mbr
🗂️ https://t.co/1EQvvtMmhi
📙https://t.co/dn3fqeTFbf
PRs welcome!
How reliably can today’s robot policies complete tasks when the world changes mid-execution?
In real-world robotics, things rarely go exactly as planned. Objects move, unexpected obstacles appear, and observations change as robots interact with their surroundings. Understanding how well robot policies handle these changes is essential for deploying Physical AI in the real world.
In our new paper, LIBERO-MAX: Do Robot Policies Adapt When the World Changes?, we introduce a benchmark designed to study this question.
We evaluate 14 state-of-the-art robot policies across 8,000 cases, introducing controlled changes after the robot has already begun acting.
The results suggest that adapting to a changing world remains a key challenge for today’s robot policies. Task success drops by 11–26 percentage points, particularly when geometry or observations change, and simply updating policies more frequently does not close the gap.
LIBERO-MAX provides a systematic way to measure these challenges, helping identify where current policies fall short and guiding the development of more adaptive and robust Physical AI systems.
📄 Paper: https://t.co/cQoDLDvgg2
🌐 Benchmark and project website: https://t.co/IeC9tN7n5j
💻 Code: https://t.co/j3JyngPD2y
Work led by @YunbeiZhang, in collaboration with Zijian Jin, Yuanzhe Liu, Janet Wang, Xilun Zhang, Yuyou Zhang, Zhenyu Zhang, Daoan Zhang, Shuaicheng Niu, Gen Li, Jianfei Yang, Jihun Hamm, Ismini Lourentzou, Weirui Ye, Bo Liu, and Peter Stone.
#Robotics #PhysicalAI #RobotLearning #RobotFoundationModels #AutonomousSystems
Life update: I recently graduated from CMU RI, and last year I went through the academic faculty job market in robot learning. I was fortunate to receive a lot of help and support from many people throughout the process.
Compared to industry job applications, there are relatively few resources available online about the faculty application process. So I decided to open-source all of my application materials from last year here:
https://t.co/4bDycVLxjO
I hope these materials can be helpful to others who are considering pursuing an academic career!
New blogpost: Fable-5.1 can hide secret messages in benign images.
We gave the image to another Fable in a fresh session and simply asked: "Is there a hidden message in this image?"
It recovered the exact message without being told how it was hidden.
Can robot foundation models handle a world that changes mid-task?
LIBERO-MAX tests:
🎯 Moving targets
🚧 New obstacles
📷 Camera shifts & more
Across 14 policies, mid-task changes reduce success by 21-56%.
If AI agents are going to run robots, they need to be tested in dynamic settings.
https://t.co/1ioqdfUQmF
🤔Simulators can generate endless robot training scenes. Which ones help the policy learn?
🤖 Meet CoDA: the data engine and the policy evolve together, driven by the policy's own failures. 🔗
📈 π0.5 on LIBERO-Pro
CoDA generates variants of the source tasks and trains π0.5 on them. After 15 rounds:
25% → 84% (position shifts)
1% → 76% (instruction changes)
+7 and +9 pts over ASPIRE after 15 rounds.
🧵👇 1/7
🔁 The memory transfers
We collected memory while training π0.5, then reused it as-is to train OpenVLA-OFT.
OOD success: 3.8% → 21.1%, with zero new environments generated.
🌱 Failures become lessons. Lessons become skills.
🧵👇6/7