Furong Huang

Assoc. prof. of CS at University of Maryland. Researcher in #PhysicalAI, #TrustworthyML, #EthicalAI, AI #Alignment, #AI for ALL.

[Furong Huang @furongh]

Hot take: robots should not dream in pixels.

Pixels are too low-level.
Latents are too opaque.

μ₀ predicts a third thing:
3D motion traces.

On real robots, it beats π₀.₅ — with ~1/100 the data scale and no action labels for world-model pretraining. 🧵

https://t.co/UfmrqNlBtw

(this video features voiceover narration)

[Yoonkyo Jung ✈️CoRL 2026 @YoonkyoJung]

I'm at #CoRL2026 in Austin this week! 🤠

Come say hi at:
1️⃣ μ0 (mu-zero): a 3D trace world model.
Pretrained on video only, it predicts 3D traces (not pixels or actions) and transfers across embodiments.
📍 Wed 11/11, 10:30 AM · Poster Session III
🔗 https://t.co/cCMloTfIwJ

[Jia-Bin Huang @jbhuang0604]

Excited for #CoRL2026 in Austin this week! 🤠

We will present mu-0, a 3D interaction-trace world model.

Key idea: Learn a "physical language” for robots purely from video. Zero action labels required to build a WM!

Code & dataset: https://t.co/GB7cq316G2

and find us Wed 11 Nov, 10:30 AM | Poster Session III

[Furong Huang @furongh]

I want robots that can watch a person perform a task, remember what matters, and adapt when the situation changes.

That’s the vision connecting our lab’s work at #CoRL2026.

A human demonstration contains much more than motion. It shows how hands and objects interact and how a task unfolds. We’re studying how to make that experience useful for robot learning—from large-scale human video in EgoScale (https://t.co/QVpzbi36ZY), to minutes of egocentric video in HumanEgo (https://t.co/VEKwhQn6YL), to a single demonstration in Dex-One2Many (https://t.co/jspMjzKDR9).

To use that experience, a robot needs to represent and remember the right things. μ₀ (https://t.co/UfmrqNlBtw) builds a world model around 3D interaction traces. What to Attend, What to Keep (https://t.co/N8Pq64GxEF) studies how skills can guide attention to vision and touch, and which events to retain as a task progresses.

Then comes acting—and responding to what happens. Fly-by-Code (https://t.co/Ow8yUShsk4) brings visual and physical feedback into embodied coding agents. Guava (https://t.co/6VVQExwgu2) explores how to transfer the capabilities of frontier vision-language models into a compact robot agent.

These are complementary steps toward robots that need less task-specific supervision, transfer knowledge across settings, and use feedback to act more reliably.

I won’t be in Austin this year, but I’m proud of the students and collaborators advancing this agenda. Ruijie Zheng is also an invited speaker at the Scaling Human2Robot Workshop!

If you’re attending CoRL, please meet the team at their talks and posters. I’d love to hear your thoughts on this direction.

Presentation details are in the posters below.

#RobotLearning #EmbodiedAI

[Furong Huang @furongh]

Such a cool work!
Quite aligned with your idea @JayLEE_0301

[Yilun Du @du_yilun]

How can a robot learn to handle delicate objects from vision?

Excited to share VisualForce, where we learn to infer force from the deformation of a soft gripper. We then train a policy to propose actions together with their predicted forces, allowing us to search for actions that apply the right amount of force.

This enables the robot to handle berries without crushing them and insert plugs without slipping, without force or tactile sensors at deployment.

https://t.co/YyNVsb4Y2g

[Jia-Bin Huang @jbhuang0604]

@furongh Thanks a lot! Yup, these are all animated from manim (https://t.co/sMxpmqf8ye). It’s fully programmable so you have full control of the animation.

[Furong Huang @furongh]

How did you make such amazing videos? @jbhuang0604!!

[Jia-Bin Huang @jbhuang0604]

How thousands of GPUs train one AI

Data parallelism helps, but not sufficient. We need new ways to split the workload along different axes.

Let's learn about 5D (data, tensor, context, pipeline, experts) parallelism and how they work together.

https://t.co/sIjxxeD4z5

[Furong Huang @furongh]

Precompute and reuse. I keep coming back to this idea, and there’s a 4 a.m. slide-making story behind it.

Back at ICLR 2023 in Kigali, Rwanda, a student of mine was helping organize the Reincarnating Reinforcement Learning workshop. A speaker couldn’t make it, and they asked whether I could give a talk.

I said yes of course, it was my student’s workshop! So I found myself making slides at 4 a.m. in my hotel room for the talk the next day. 😂

My talk was “Adaptable Reinforcement Learning in an Ever-Changing World: Reuse to Learn and Adapt Efficiently.” I also joined the panel, “Challenges & Open Problems in Reusing Prior Computation.”

It was absolutely a great workshop. What stayed with me was a simple question: why keep starting over when we’ve already spent so much compute learning something useful?

Today, that sounds almost too obvious. Isn’t that a big part of what foundation models are about? And really, RL post-training follows the same logic: start with a pretrained model, reuse what it has learned, and build on it with RL.

But back then, training a specialist RL agent from scratch for each task was still very common. That workshop helped me put words to something that has become a recurring theme in our research.

Once you take reuse seriously, the interesting question becomes: what should we learn ahead of time, and what should we leave for adaptation when the actual task arrives?

You can see that question across our work, just to name a few:
• SMART: pretrain control representations that transfer across tasks.
• DynaFLIP: learn dynamics-aware perception that different robot policies can build on.
• μ₀: learn a world model from video and reuse it across robot embodiments.
• FlowBank: precompute useful agentic workflows, then choose one for each query.

And now, Meta²: precompute and reuse knowledge about how to improve.

An agent’s harness includes the prompts, tools, memory, hooks etc around the model. Often, you only discover what needs changing once the agent is already working. A missing tool. An instruction that keeps being forgotten. A check that needs to happen repeatedly.

We want the agent to adapt as those needs appear. But reasoning about every possible change from scratch is expensive, and making changes doesn’t automatically mean making progress.

So we borrow a core idea from meta-learning: learn how to adapt.

Offline, Meta² learns a textual Meta-Skill that guides an evolver on when and how to change the harness, including when to leave it alone. We learn this guidance by comparing runs with no evolution, runs with the current evolver, and training runs where the evolver gets privileged verifier feedback.

During a new run, that learned guidance stays fixed. At checkpoints, the evolver combines it with evidence from the current run to decide whether to keep or change the harness. The underlying model stays frozen.

The guidance gets reused. The actual edits depend on what happens.

One result I especially like: carrying one task’s harness edits into future tasks actually hurt performance in our experiments. Knowing how to adapt transferred better than keeping the adaptations themselves.

That distinction matters for RSI. A system that keeps improving needs to figure out what to preserve from experience, what to revise, and what was only useful in one particular situation.

Meta² gives us one concrete piece of that: reusable guidance for improvement, paired with adaptation while the agent is still working.

Longer term, I see RSI as potentially the co-evolution of models, harnesses, and the environments they learn in. Progress in each could help the others improve.

For now, we’re nailing down the harness part! I’m really proud of my students and collaborators for finding an efficient and effective way to make this work.

That 4 a.m. slide-making session stayed with me longer than I expected. 🧬

[Lingzhi Yuan @lingzhi_yuan]

Is your agent's harness frozen before the run even starts?

In the push toward recursive self-improvement (RSI), a system that can't adjust to new evidence mid-run isn't truly adaptive. To keep evolving online, it needs guidance on how and when to evolve.

Introducing Meta²: It learns this evolution guidance offline and applies it dynamically within every single run.
🧬👇
https://t.co/1assuQPPYN
https://t.co/4qo0y2UXYI

[Furong Huang @furongh]

Precompute and reuse. I keep coming back to this idea, and there’s a 4 a.m. slide-making story behind it.

Back at ICLR 2023 in Kigali, Rwanda, a student of mine was helping organize the Reincarnating Reinforcement Learning workshop. A speaker couldn’t make it, and they asked whether I could give a talk.

I said yes of course, it was my student’s workshop! So I found myself making slides at 4 a.m. in my hotel room for the talk the next day. 😂

My talk was “Adaptable Reinforcement Learning in an Ever-Changing World: Reuse to Learn and Adapt Efficiently.” I also joined the panel, “Challenges & Open Problems in Reusing Prior Computation.”

It was absolutely a great workshop. What stayed with me was a simple question: why keep starting over when we’ve already spent so much compute learning something useful?

Today, that sounds almost too obvious. Isn’t that a big part of what foundation models are about? And really, RL post-training follows the same logic: start with a pretrained model, reuse what it has learned, and build on it with RL.

But back then, training a specialist RL agent from scratch for each task was still very common. That workshop helped me put words to something that has become a recurring theme in our research.

Once you take reuse seriously, the interesting question becomes: what should we learn ahead of time, and what should we leave for adaptation when the actual task arrives?

You can see that question across our work, just to name a few:
• SMART: pretrain control representations that transfer across tasks.
• DynaFLIP: learn dynamics-aware perception that different robot policies can build on.
• μ₀: learn a world model from video and reuse it across robot embodiments.
• FlowBank: precompute useful agentic workflows, then choose one for each query.

And now, Meta²: precompute and reuse knowledge about how to improve.

An agent’s harness includes the prompts, tools, memory, hooks etc around the model. Often, you only discover what needs changing once the agent is already working. A missing tool. An instruction that keeps being forgotten. A check that needs to happen repeatedly.

We want the agent to adapt as those needs appear. But reasoning about every possible change from scratch is expensive, and making changes doesn’t automatically mean making progress.

So we borrow a core idea from meta-learning: learn how to adapt.

Offline, Meta² learns a textual Meta-Skill that guides an evolver on when and how to change the harness, including when to leave it alone. We learn this guidance by comparing runs with no evolution, runs with the current evolver, and training runs where the evolver gets privileged verifier feedback.

During a new run, that learned guidance stays fixed. At checkpoints, the evolver combines it with evidence from the current run to decide whether to keep or change the harness. The underlying model stays frozen.

The guidance gets reused. The actual edits depend on what happens.

One result I especially like: carrying one task’s harness edits into future tasks actually hurt performance in our experiments. Knowing how to adapt transferred better than keeping the adaptations themselves.

That distinction matters for RSI. A system that keeps improving needs to figure out what to preserve from experience, what to revise, and what was only useful in one particular situation.

Meta² gives us one concrete piece of that: reusable guidance for improvement, paired with adaptation while the agent is still working.

Longer term, I see RSI as potentially the co-evolution of models, harnesses, and the environments they learn in. Progress in each could help the others improve.

For now, we’re nailing down the harness part! I’m really proud of my students and collaborators for finding an efficient and effective way to make this work.

That 4 a.m. slide-making session stayed with me longer than I expected. 🧬

[Lingzhi Yuan @lingzhi_yuan]

Is your agent's harness frozen before the run even starts?

In the push toward recursive self-improvement (RSI), a system that can't adjust to new evidence mid-run isn't truly adaptive. To keep evolving online, it needs guidance on how and when to evolve.

Introducing Meta²: It learns this evolution guidance offline and applies it dynamically within every single run.
🧬👇
https://t.co/1assuQPPYN
https://t.co/4qo0y2UXYI

[Furong Huang @furongh]

Try it: pip install icalens
https://t.co/Q44rPhcJMF

Website: https://t.co/wTQiRbDZ5V
GitHub: https://t.co/fpW8YaktcQ
Paper: https://t.co/UN0DsdcZej

[Feijiang Han😈 @feijianghan]

check out this intuitive video on why interpretable directions in llms look non-gaussian, how ica finds them in minutes per layer, and the evaluation bug that made ica look weak for 3 years

[Feijiang Han😈 @feijianghan]

check out this intuitive video on why interpretable directions in llms look non-gaussian, how ica finds them in minutes per layer, and the evaluation bug that made ica look weak for 3 years

[Feijiang Han😈 @feijianghan]

Here’s a concrete example of why I like combining ICA Lens with Logit/J/R Lens.
(by @liusida2007)

Take “The capital of France is” in Qwen3.5-2B. At the final token, directly decoding the layer-17 post-block residual with Logit Lens, J-Lens, or R-Lens gives mostly underscore tokens.

Decompose that same activation with ICA Lens, then inspect the components with those lenses. The top two components have clear city-related and France-related readouts, as shown below.

Directly decoding the full residual gives a readout of the combined vector. ICA separates it into components we can inspect individually.

That’s the value of combining these tools: structure that’s hard to see in the full activation can become much clearer after decomposition.

[Furong Huang @furongh]

ICA Lens is open-source, lightweight, and requires no training of another sparse dictionary. 🌹 Try it yourself: pip install icalens

https://t.co/Q44rPhcJMF

[Furong Huang @furongh]

SAEs have become the dominant approach in mechanistic interpretability. But what if a good old statistical method from decades ago could do just as well, at a fraction of the cost?

In our latest work, we revisited Independent Component Analysis (ICA) and discovered something surprising: a bug in the evaluation code of an influential early SAE paper had significantly underestimated ICA's interpretability! After correcting it, we found ICA to be remarkably competitive with SAEs.

Many of my academic friends from the good old days probably remember me as the tensor decomposition and spectral methods person. So this feels a little like going back to my roots. :) Sometimes, old mathematical tools still have surprises in store!

But there's another motivation behind this work that I care about even more.

I've always wished mechanistic interpretability could be more actionable. Understanding what happens inside a neural network is fascinating, but what comes next? Can we use that understanding to actually intervene, steer behavior, and improve the model?

With ICA Lens, we take steps in that direction, from discovering and annotating interpretable features to using them to understand new hidden states and steer model behavior.

Interpretability shouldn't end with understanding. It should empower us to act.

And we'd love for the community to try it! ICA Lens is open-source, lightweight, and requires no training of another sparse dictionary.

🌹 Try it yourself:
pip install icalens
https://t.co/5jaTp29eX9

Curious to hear what the mechanistic interpretability community thinks!

[Niloofar @niloofar_mire]

@furongh @BergKirkpatrick @humansand Thanks so much Furong! Great point! So I would say its more the former. We dont do any “bespoke” modifications or any forcing. We put the model in a collaborative task/environment and scale rl and the rollouts and the optimization pressure takes care of the rest!

[Furong Huang @furongh]

SAEs have become the dominant approach in mechanistic interpretability. But what if a good old statistical method from decades ago could do just as well, at a fraction of the cost?

In our latest work, we revisited Independent Component Analysis (ICA) and discovered something surprising: a bug in the evaluation code of an influential early SAE paper had significantly underestimated ICA's interpretability! After correcting it, we found ICA to be remarkably competitive with SAEs.

Many of my academic friends from the good old days probably remember me as the tensor decomposition and spectral methods person. So this feels a little like going back to my roots. :) Sometimes, old mathematical tools still have surprises in store!

But there's another motivation behind this work that I care about even more.

I've always wished mechanistic interpretability could be more actionable. Understanding what happens inside a neural network is fascinating, but what comes next? Can we use that understanding to actually intervene, steer behavior, and improve the model?

With ICA Lens, we take steps in that direction, from discovering and annotating interpretable features to using them to understand new hidden states and steer model behavior.

Interpretability shouldn't end with understanding. It should empower us to act.

And we'd love for the community to try it! ICA Lens is open-source, lightweight, and requires no training of another sparse dictionary.

🌹 Try it yourself:
pip install icalens
https://t.co/5jaTp29eX9

Curious to hear what the mechanistic interpretability community thinks!

[Feijiang Han😈 @feijianghan]

We revisited Independent Component Analysis (ICA), a decades-old statistical method, and found that it works far better for interpreting LLM activations than we expected!
This old baseline deserves a second look.

Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find ICA is actually competitive with the public SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7–43× less parameter memory.

We're releasing ICA Lens, which lets you quickly discover, annotate, and evaluate interpretable directions in LLM residual-stream activations. No need to train another dictionary!

More broadly, our results suggest that non-Gaussianity is a useful inductive bias for efficiently finding interpretable directions.

Try it: pip install icalens https://t.co/vKYFk0ubna
Paper: https://t.co/RBTPx3SPLp
1/8

[Furong Huang @furongh]

Amazing work! Curious about the collaboration aspect. Is it because the simulations involve collaboration, or is the model directly trained on collaborative data?

I guess the key distinction is that, in the former, collaboration is used to generate feedback signals, whereas in the latter, collaborative behavior is directly imitated. Is that a fair interpretation?

[Furong Huang @furongh]

This is timely! Building the data engine for robotics is crucial to combat the “small data” problem in physical intelligence.

[Jim Fan @DrJimFan]

We built a robotics data engine with Astra-level performance at 125x lower cost. And we are open-sourcing the weights today! Introducing Flora, a suite of open models to parse the physical world and turn raw footage into rich training signals.

Flora already powers our in-house data engine, labeling 200K hours of videos a day on 500 GPUs. We created it to scale the frontier for robot learning. Now you own the weights too.

Flora gives you:

- High-quality, timestamped captions to curate and filter human egocentric and robot data;
- Dense progress rewards for quality control and RL post-training;
- Rubrics scoring for real-world benchmarking

Try it today, links in thread:

[Furong Huang @furongh]

Cool work!

[Zhenting Qi @ZhentingQi]

Research index (e.g. Google Scholar) is one of the largest collaborative systems where millions of human users build on each other's work across centuries.

How about we build something similar for async LLM agents?

We give agents a minimal hypertextual system. They asynchronously search for and read related work, cite the work that they find useful, and publish (mini-)papers.

Each search returns only the top k papers, ranked by relevance and a PageRank-like citation weight, which bounds the read cost of each agent to O(k), independent of agent count.

We show that performance effectively scales as we keep adding agents, and the system is robust to absence/failures of individual agents.

[Furong Huang @furongh]

Congratulations @canondetortugas!

[Dylan Foster 🐢 @canondetortugas]

A super exciting announcement: In January 2027, I'll be joining @UCBerkeley as an assistant professor in EECS and Statistics.

I'll be recruiting PhD students and postdocs this cycle, starting Fall 2027. If you're interested the science of RL and deep learning, please apply.