Zekun Wang

🎓 Ph.D. @GeorgiaTech | M.S. @UMich | B.S. @penn_state
🔬 Continual Learning | Compositional Generalization | Generative Models | Unsupervised Learning

[Zekun Wang @zekunwang28]

As more context arrives, SCoL learns which layers to update to absorb new information while limiting forgetting.

Check out our latest work: Self-Consolidating Language Models
https://t.co/CMqfIzygrb
1/7

[Zekun Wang @zekunwang28]

Task 1 also recovers quickly while the model keeps learning later tasks. On CW10, Trust Region restores 90% of Task 1's initial success in just 2 updates while training on Task 10, compared with 8,000 for Replay.
https://t.co/mYzNtcRIKr

[Zekun Wang @zekunwang28]

We're happy to share that “Trust Region Continual Learning as an Implicit Meta-Learner” was accepted to NeurIPS 2026!

It connects continual and meta-learning in diffusion models.

After 10 tasks, the robot still remembers how to use a hammer.

[Zekun Wang @zekunwang28]

Continual learning needn't mean catastrophic forgetting. After 10 robot tasks, Trust Region reaches 88.3% average success, with forgetting down from 8.2 to 4.4 percentage points vs. Replay. Both image benchmarks also improve in quality and retention.
https://t.co/lFFgjZtvR7

[Zekun Wang @zekunwang28]

Here's the blog, with interactive explanations of rank-1 Fisher, replay, trust regions, and the connection to meta-learning.
https://t.co/zV5gHIVWJg

[Zekun Wang @zekunwang28]

7/7
Our goal is for models to keep learning after deployment, with each new experience building on what came before.

Joint work with Anant Gupta, Zihan Dong, and Christopher J. MacLellan.

Interactive explanation and full paper:
https://t.co/CMqfIzygrb

[Zekun Wang @zekunwang28]

As more context arrives, SCoL learns which layers to update to absorb new information while limiting forgetting.

Check out our latest work: Self-Consolidating Language Models
https://t.co/CMqfIzygrb
1/7

[Zekun Wang @zekunwang28]

6/7
The learned selections also overlap with layers that Fisher information identifies as sensitive to the incoming context.

We measure this after training. The policy never sees Fisher scores, so the alignment offers a clue about what it has learned.

[Zekun Wang @zekunwang28]

5/7
With Qwen2.5-7B, adding the forgetting penalty raises final retention accuracy from 14.64% to 20.46% over a stream of 100 SQuAD passages.
On LongBench v2, the policy also transfers to contexts up to 64k tokens after training with up to 32k.

[Zekun Wang @zekunwang28]

4/7
We use meta RL to learn these choices.

Candidate updates earn reward for learning the current chunk while preserving earlier knowledge.

The best update continues the stream. Feedback across the stream trains the model to make better choices next time.

[Zekun Wang @zekunwang28]

3/7
The catch is forgetting. Each new update can damage what the model just learned.
SCoL uses sparse LoRA updates, with the model itself choosing which layers to change for each chunk.

Those choices need to balance learning new information with retaining the old.

[Zekun Wang @zekunwang28]

2/7
The intuition comes from human memory. A conversation can change what we know long after the words are gone.

We want context to leave that kind of lasting trace in an LLM. This turns long-context inference into a continual learning problem.

[Zekun Wang @zekunwang28]

Excited to co-organize the NeurIPS 2026 Workshop on Towards Test-Time Continual Learning Agents!

Looking forward to bringing together the continual learning, agent, robotics, and LLM communities to discuss how AI can keep learning after deployment! Hope to see your submissions!

[Brian Zheyuan Zhang @zheyuanzhang99]

📢 Announcing Towards Test-Time Continual Learning Agents (TTCL) workshop at NeurIPS 2026
#NeurIPS #NeurIPS2026

How can AI agents keep learning after deployment without catastrophic forgetting?

🗓 Submission deadline: Aug 29, 2026 AoE
🔗 https://t.co/Kc6tfejUEW

🧵1/n

[Zekun Wang @zekunwang28]

Diffusion doesn’t just help replay, it also gives a computable Fisher that makes consolidation (EWC) finally work the way we hoped.

[Zekun Wang @zekunwang28]

Continual learning has a “holy grail”: EWC works if you have a good Fisher. But Fisher is usually too expensive to estimate well at scale.
Our ICLR 26 paper shows: diffusion models naturally reveal a clean Fisher structure, making EWC practical + strong
https://t.co/L9yAEBQn8R

[Zekun Wang @zekunwang28]

Result: plug this into continual image generation task, and forgetting (measured by FID) collapses across MNIST/FMNIST/CIFAR-10/ImageNet-1k.
Example: ImageNet-1k forgetting 46 → 15, MNIST: 2.3 → 0.6, FashionMNIST: 3.9 → 0.9, CIFAR10: 16.6 → 7.4.

[Zekun Wang @zekunwang28]

Method: rank-1 EWC (cheap like diagonal: only using gradients, but captures the true dominant curvature direction) + generative replay (reduces replay drift + encourages shared optima).
Replay keeps shared optimal available; rank-1 EWC prevents destructive updates on old tasks.

[Zekun Wang @zekunwang28]

Key finding: in diffusion training, low-SNR timesteps make per-sample gradients strongly collinear → the empirical Fisher is effectively rank-1 (dominated by one direction).
So Fisher stops being “mysterious curvature” and becomes something you can compute reliably.

[Zekun Wang @zekunwang28]

Our NAACL paper has been adopted as a baseline for BabyLM 3.0!!!!!

Looking forward to seeing how the community builds on it!!!!! @ziqiao_ma @babyLMchallenge

[babyLM @babyLMchallenge]

We are expecting🫄
A 3rd BabyLM👶, as a workshop @emnlpmeeting
Kept: all
New:
Interaction (education, agentic) track
Workshop papers
More in 🧵

[Zekun Wang @zekunwang28]

Check out our latest research on language acquisition! Tackling this big mystery is by far the project I am most proud of!

Shout out to my mentor @ziqiao_ma, he's simply the best! And a great appreciation for Prof. Joyce Chai for her value inputs!

[Martin Ziqiao Ma @ziqiao_ma]

Happy to share some “slow research” - it's been 15 months and it's now (finally) on arXiv! Our language development is different from LLMs. We're asking: How do you interactively babysit a language model from scratch, and would it help?🤔
🔗 https://t.co/OyHqBSShaQ
@Michigan_AI

[Zekun Wang @zekunwang28]

@ziqiao_ma @ICatGT @ChrisJMacLellan thanks!!

[Zekun Wang @zekunwang28]

As my first tweet, I’m thrilled to share that I will join @ICatGT this fall as a PhD student working with Prof. @ChrisJMacLellan !! We will be investigating human-like and human-level learning for AI systems. Can’t believe this dream came true!