Haoyu Zhou

PhD @a2i_oxford, I work on robotics.

[Haoyu Zhou @object814]

Thanks to my wonderful co-authors @JoeMWatson and @AnsonISL , and my supervisor @IngmarPosner , at the Applied AI Lab @a2i_oxford , Oxford Robotics Institute @oxfordrobots , University of Oxford.

Please feel free to check out more details on our paper and website!

[Haoyu Zhou @object814]

Our factorisation shows difficulty rises from action to perception to full composition📈, matching where each modality enters the model:
💪action hits only the dynamics;
👀perception shifts the encoder, then drifts the dynamics;
💫full composition directly hits both.
9 / N

[Haoyu Zhou @object814]

Takeaways:
Reuse without forgetting remains open for world models🔓, and we think progress lies in:
🧠 stable, general visual representations learnt continually;
🧩 modular designs that explicitly encourage composition.
Our benchmark is built to measure exactly that📏
10 / N

[Haoyu Zhou @object814]

We also build on a MoE world model (PWM) for continual learning: new dynamics experts per task ➕, old ones frozen ❄️, and a router learning to combine them🔀
This modular design pushes the reuse-retention frontier🚀, but only with a privileged pretrained encoder⚠️
8 / N

[Haoyu Zhou @object814]

We first test state-of-the-art monolithic world models (DreamerV3 & TD-MPC2) with conventional CL methods🧪: ER, EWC, and PackNet.
None achieves both reuse and retention, each trades one for the other⚖️
7 / N

[Haoyu Zhou @object814]

How do you apply continual learning to a world model? We split it into:
🔁 Task-agnostic backbone capturing how the world evolves, learnt continually;
🔀 Task-specific heads defining each task's objective, trained fresh per task.
CL methods act only on the backbone.
6 / N

[Haoyu Zhou @object814]

To pinpoint where reuse breaks, we factorise composition along the two inputs of a world model as shown above:
💪 Action: new motion combos, same scene
👀 Perception: same motion, new visual combos
💫 Full: both

6 task suites in total, built on Meta-World assets.
5 / N

[Haoyu Zhou @object814]

Our fix🔧: Each curriculum in our benchmark ends with a composition task built by recombining the primitives before it.
Forward transfer on this final task then measures reuse more directly🎯, while evaluating earlier tasks tracks forgetting😃.
4 / N

[Haoyu Zhou @object814]

But existing benchmarks can't measure reuse cleanly⚠️
Every new task brings unseen content alongside what recurs, so adaptation efficiency mixes two abilities: reusing prior knowledge🧩; and simply learning fast⚡. Good forward transfer score can't tell you which happened🤷
3 / N

[Haoyu Zhou @object814]

Continual learning asks agents to adapt to new tasks without forgetting old ones🤖
We think the scalable way to do this in robotics is through reuse without forgetting🧩: recombining and reusing previous knowledge, rather than relying on new capacity for every task.
2 / N

[Haoyu Zhou @object814]

How well do world models reuse what they've already learnt, without forgetting it? 🤔

I am excited to introduce our new work that looks into this! 💡

Website: https://t.co/qcI3lscrXn
Paper: https://t.co/w0N1VcOYBt
1 / N👇

#robotics #machinelearning #worldmodel

[Haoyu Zhou @object814]

@russwest44 Thank you russ

[Weikun Peng @PengWeikun]

🎉Our new work, "iTACO: Interactable Digital Twins of Articulated Objects from Casually Captured RGBD Videos" got accepted to 3DV 2026!

💡Existing methods for reconstructing articulated objects often require significant human effort to align all views to the same coordinate system. We instead propose building digital twins of articulated objects from casually captured RGBD videos, which is much more practical and scalable.

💡We therefore propose iTACO, a training-free pipeline for building digital twins of articulated objects from casually captured RGBD videos.

💡At the same time, we build a large dataset for robust benchmarking, which contains 20x more objects than existing datasets.

🪩The full dataset and source code are released. Visit our website for more details: https://t.co/NBTGBDeQVF