Yijun (Joanna) Liang

CS PhD Student @umdcs @UMIACS

[William Yijiang Li @Williamiumli]

🚀 New paper alert: On-Policy Self-Distillation without Any Supervision !!

🧠 Can a model truly on-policy “self-distill"?

On-policy self-distillation removes the need for a stronger teacher, but still relies on an external ground-truth solution to make the self-teacher more capable.

Our new work asks: can we remove supervision altogether?

Can an LLM improve its reasoning entirely from its own generations—without ground-truth answers, verifiers, or a stronger teacher?

♻️ We introduce U-OPSD, a simple approach toward genuine self-distillation:

For each problem, the model samples multiple solutions from itself:

🗳️ Agreement identifies a likely solution through majority vote
🔀 Disagreement reveals trajectories where the model may be going wrong
🎓 The pseudo solution conditions the self-teacher, which provides dense token-level guidance along those disagreeing trajectories

U-OPSD turns self-consistency into supervision: agreement builds the teacher context, while disagreement determines where to distill.

🔓 No external supervision at all

📚 Unlabeled problems only:
🚫 GT solutions
🚫 Verifier
🚫 External strong teacher
🚫 Environment feedback

📊 Yet, across 5 math reasoning benchmarks and 6 Qwen3 settings, U-OPSD consistently improves the base model and matches or surpasses supervised SFT, GRPO, and OPSD.

🌙 Non-thinking mode:
📈 +8.5 / +10.7 over the 4B / 8B base models
🏆 +3.2 / +2.3 over GT-supervised OPSD
🚀 +7.0–11.3 over label-free self-rewarding RL methods including TTRL, RENT, and Intuitor

🧠 Thinking mode:
📈 +2.2 / +1.9 over the 4B / 8B base models
🤝 On par with GT-supervised OPSD and GRPO
🚀 +0.8–1.4 over label-free self-rewarding RL methods including TTRL, RENT, and Intuitor

Meet U-OPSD 👇
📄 ArXiv Paper: https://t.co/sGBO2uT8DQ
💻 GitHub: https://t.co/T5eZtbUlYJ
🌐 Project Page: https://t.co/7oDlOLCPK2
🤗 Hugging Face Paper: https://t.co/hWWVJoISOw
(Plz upvote if you can !!

Amazing collaboration with @ Bingyang @JoLiang17 @tian_yunjie @Di Fu and Nuno !
Grateful for all the insightful discussions and exploration together!

#LLM #OPD #OPSD #Distillation

[Yijun (Joanna) Liang @JoLiang17]

🔍Do we really need to give models more information to help them learn?

Recent on-policy self-distillation methods rely on answer hints, reasoning traces, or visual evidence to construct a better learning signal.
We wondered:
❓What if none of them were necessary?

🔄We introduce Visual Contrastive Self-Distillation (VCSD), a simpler form of OPSD driven purely by input conditioning.

Instead of adding supervision, we compare the same EMA teacher under two matched visual conditions:
🖼️ Original image
⬛ Content-erased control

Their prediction difference reveals what the image actually contributes, and becomes the self-distillation signal. By suppressing shared language priors and emphasizing visually supported tokens, VCSD rebalances the distillation target toward the visual modality.
✅ No external teacher
✅ No answer hints
✅ No reasoning traces
✅ No visual evidence
✅ No additional inference-time cost

📈 Across 6 model scales and 7 vision-language benchmarks, VCSD consistently outperforms matched OPSD, with gains of up to +4.5 points.

Meet VCSD 👇
🌐 Project page: https://t.co/pIFQI0a1vL
📄 Huggingface Paper: https://t.co/MxaYrk1Lrb
Amazing collaboration with @tian_yunjie , @Williamiumli , @Yuqi_Jia7 , @zhoutianyi , @furongh And Di Fu!
Grateful for all the insightful discussions and exploration together!
@umdcs @ucsd_cse

[Yijun (Joanna) Liang @JoLiang17]

Excited to share this work that I've been fortunate to collaborate on!

We propose a scalable way to collect training-time reward-hacking trajectories and show that they lead to substantially stronger monitors for detecting real inference-time reward hacking.

Big thanks to all my collaborators—this was a pleasure to work on🎉

[TurningPoint AI @TurningPointAI]

Many monitors are trained & evaluated on prompt-elicited hacking trajectories, where models are explicitly asked to exploit the reward signal. But the real test is whether they catch the training-time hacks that naturally emerge during RL training without hacking instructions.

❗️We find a large mismatch:
📉 Detection accuracy: 97.1% on prompted hacks → BUT 28.0% on training-time hacks.

🔍 To study this, we introduce Trace-and-Amplify (TA), a framework for collecting training-time reward-hacking trajectories at scale without hacking instructions.

✅ Monitors trained on TA-collected trajectories perform much better on real inference-time hacks, and generalize better to held-out hacking types:
Detection accuracy 59.98% (PE-trained) → 90.16% (TA-trained).

🌐 Project: https://t.co/unI5L32fRH
📄 Paper: https://t.co/Q7NXjg1ERU

Joint work with @LilichenLi3146, @hgzhou42, @JoLiang17, @zhoutianyi, and @cho_jui_hsieh at PKU, UCLA, and UMD
@PKU1898 @umdcs @UCLACS
Grateful for everyone’s contributions and support!

[TurningPoint AI @TurningPointAI]

Many monitors are trained & evaluated on prompt-elicited hacking trajectories, where models are explicitly asked to exploit the reward signal. But the real test is whether they catch the training-time hacks that naturally emerge during RL training without hacking instructions.

❗️We find a large mismatch:
📉 Detection accuracy: 97.1% on prompted hacks → BUT 28.0% on training-time hacks.

🔍 To study this, we introduce Trace-and-Amplify (TA), a framework for collecting training-time reward-hacking trajectories at scale without hacking instructions.

✅ Monitors trained on TA-collected trajectories perform much better on real inference-time hacks, and generalize better to held-out hacking types:
Detection accuracy 59.98% (PE-trained) → 90.16% (TA-trained).

🌐 Project: https://t.co/unI5L32fRH
📄 Paper: https://t.co/Q7NXjg1ERU

Joint work with @LilichenLi3146, @hgzhou42, @JoLiang17, @zhoutianyi, and @cho_jui_hsieh at PKU, UCLA, and UMD
@PKU1898 @umdcs @UCLACS
Grateful for everyone’s contributions and support!

[Yijun (Joanna) Liang @JoLiang17]

🔍 Before an AI can explore the world, it must learn what questions are worth asking.
❓ “What color is the fireplace?”
➡️ “Is the object on top of the fireplace shown in the mirror?”
🖼️ Same image. Completely different supervision.
Only one forces a model to inspect, ground, reason, and explore.

Today's Vision-Language Models (VLMs) are trained primarily as answerers. But intelligent systems should also know how to ask good questions.
Most VLMs learn from fixed question distributions. We let the question distribution evolve.

We introduce Self-Evolving Visual Questioner (SeeQ), a framework that enables a VLM to continuously improve itself as a visual questioner using its own generated QA data.
✅ No human annotations
✅ No external teacher models
✅ No reward models

⚠️ A key challenge is collapse: without careful control, self-generated questions quickly converge toward repetitive and low-information patterns.

🔄 Our self-evolving proposal → rewrite → filter loop maintains exploration while continuously pushing the model toward broader, harder, and more visually grounded questions.

📏 To measure this evolution, we built an evaluation framework for questions that demand:
🔎 Harder visual search
🎯 Tighter grounding
🧠 Deeper contextual and spatial reasoning
🌈 Greater diversity

📈 After just two rounds of self-evolution:
⬆️ Question quality improves by 82%
💪 QA performance is preserved on VStar, CVBench, and RealWorldQA

Meet SeeQ 👇
🌐 Project page:
https://t.co/a0HIK2Qwep
📄 Arxiv:
https://t.co/RiEoo0cQI1
💻 Codebase:
https://t.co/8cvbxqzKpJ

Many thanks to the collaborators Hengguang @hgzhou42 , @Ming_Liiii , Lichen Li, @cho_jui_hsieh and @zhoutianyi at UMD and UCLA @umdcs @UCLACS

[Furong Huang @furongh]

Hot take: robots should not dream in pixels.

Pixels are too low-level.
Latents are too opaque.

μ₀ predicts a third thing:
3D motion traces.

On real robots, it beats π₀.₅ — with ~1/100 the data scale and no action labels for world-model pretraining. 🧵

https://t.co/UfmrqNlBtw

(this video features voiceover narration)

[Shweta Bhardwaj @sh10bhardwaj]

🚀 Presenting our work on 21st Oct at #ICCV2025 Honolulu! 🍍🏝️ 🌺 Find us at Poster Exhibit Hall-1 #151 or DM to chat

Diffusion Curriculum: Synthetic-to-Real Data Curriculum via Image-Guided Diffusion

(1/3)
🌐 Website: https://t.co/AAOyZHtYJe
📄 Paper: https://t.co/I5AdMA6AC5

[Yijun (Joanna) Liang @JoLiang17]

🤝 Open to #Collaboration and #Internship Opportunities on:
🧠 Data-centric AI
🤖 Vision-language Model training and evaluation
Shoutout to amazing co-authors @sh10bhardwaj @zhoutianyi!
#VisionLanguageModels #DCAI #DataCentric #ResponsibleAI #ICCV #AI #ML #ComputerVision
(3/3)

[Yijun (Joanna) Liang @JoLiang17]

🚀 Presenting our work on #ICCV2025 Honolulu! 🍍🏝️🌺

Diffusion Curriculum: Synthetic-to-Real Data Curriculum via Image-Guided Diffusion

✔️Poster: Oct 21 (Tue) Session 1 & Exhibit Hall, #151
🌐 Website: https://t.co/IJTJq1UIpy
📄 Paper: https://t.co/d8Sui81IIS
(1/3)

[Yijun (Joanna) Liang @JoLiang17]

🌍Real-world data is noisy, imbalanced & scarce → brittle models
🎨T2I synthesis boosts diversity but suffers from a large syn-to-real gap
✨DisCL leverages a spectrum of syn-to-real data with image guidance and curriculum to progressively bridge this gap
(2/3)

[Tianyi Zhou @zhoutianyi]

🎨Color plays a critical role in human perception and reasoning in daily applications, e.g., medical test kit readings, shopping, wild animal study, and agriculture.
But is it so for existing #AI and #VLMs? We propose🔥ColorBench, which exposes weaknesses of existing🤖VLMs👇