Huan-ang Gao

Ph.D. student @ Tsinghua CS | AutoResearch & Agentic RL & self-evolving agents

[zhilong zhang @zhilong_zhang26]

RL post-training works incredibly well, but its parameter update is still a black box.

🧐Our question: can we understand and locate the effective part of an RL update, remove the noisy directions, and even improve the model after training?

We find the answer is yes.

Excited to share our new work:
https://t.co/1frMj9HB5c
Joint work with ：
@hello_gensi @huiyeruzhou @c7wc7w @Han_lin_Wu @kb_syx @yaqinzhang @haozhou_ai

📌The reasoning-effective part of an RL update can be represented as a compact rewiring matrix in the base model’s spectral space.

🤖 This leads to SAR: a training-free post-hoc method that projects the raw RL update onto this compact reasoning core, enabling us to understand, purify, and merge RL-trained models.

Key results:
1️⃣ The reasoning core of RL is highly compact:
With less than 1% spectral parameters, SAR can recover or improve full-RL gains.

2️⃣ Dropping noisy directions helps:
For math, SAR solves the exploration degradation often observed after RL. It also improves agentic coding on large-scale in-house models.

3️⃣ SAR purifies mixed-domain RL:
On a 32B model jointly trained for math, code, instruction following, and chat, SAR improves coding and math exploration while keeping instruction following stable.

4️⃣ SAR makes model merging stronger:
After SAR purification, merged models can surpass the best single-domain experts. We observe the same trend on production-scale in-house models.

Overall, SAR is training-free, broadly useful, and gives us a new geometric lens on what RL is really changing inside reasoning models.

[Huan-ang Gao @c7wc7w]

[5/n]

Paper:
https://t.co/bci3G7BrxM
Project page:
https://t.co/peZTH4PXTK
Code:
https://t.co/UOw9Rf8QRW
Models:
https://t.co/t6lG92bizq

I was fortunate to help lead this project with an amazing team from SIA-Lab, joint lab of Tsinghua AIR and Bytedance Seed.

Feedback, questions, and discussion are very welcome. If you find the work useful, we’d also appreciate your support on Hugging Face 🤗

[Huan-ang Gao @c7wc7w]

RLVR is powerful, but repeating it for every larger target model is expensive: each target must generate its own rollouts and rediscover useful learning signals from sparse outcome rewards.

Can RL on a small, weaker model improve a stronger student—even when the student already outperforms the small model after RL?

We found that it can—but not by distilling the weak model itself.

Today, we’re excited to share Direct-OPD, joint work w/ @Shiyuan040223, @c7wc7w, @Ahydchh, @Han_lin_Wu, @zhilong_zhang26, Zheng Jiang, @HBX_hbx, Wei-Ying Ma, @yaqinzhang, @haozhou_ai, developed at SIA-Lab @hello_gensi, a joint lab of Tsinghua AIR and ByteDance Seed.

https://t.co/peZTH4QvJi

Our alternative is simple:

1. Run RL on a small model, where exploration and rollouts are cheaper.
2. Treat the model’s pre- and post-RL checkpoints as a teacher pair, whose difference captures the direction learned through RL.
3. On-policy distill this policy shift—what RL changed—using the stronger student’s own rollouts.

In one setting on AIME24:
[*] 1.5B teacher pair: pre-RL and post-RL checkpoints (Post-RL teacher score: 51.3)
[*] 7B student before transfer: 56.7
[*] 7B student + vanilla on-policy distillation: ~50
[*] 7B student + Direct-OPD: 63.1 (+6.4)

The 7B student already starts stronger than the post-RL teacher. Distilling the teacher itself makes the student worse. But distilling what the teacher pair learned through RL improves it further.

In other words, the reusable outcome of an RL run is not only the final checkpoint—it can also be the policy shift encoded by the pre- and post-RL checkpoint pair.

[Huan-ang Gao @c7wc7w]

[1/n]
Why doesn’t ordinary distillation work here?

A post-RL teacher contains two things at once:
[*] Useful changes introduced by RL
[*]The smaller model’s original preferences and capacity limits

When the student is already stronger, matching the teacher’s full distribution can overwrite the student’s better behavior.

That is exactly what we observe:
R1-Distill-7B starts at 56.7, already above the JustRL-1.5B teacher at 51.3.
Vanilla on-policy distillation pulls the 7B student down to roughly 50.