Haohan Chi

Incoming PhD at Tsinghua AIR GenSI @hello_gensi, exploring how models learn more reliably from feedback, trajectories, and structured experience

[Huan-ang Gao @c7wc7w]

RLVR is powerful, but repeating it for every larger target model is expensive: each target must generate its own rollouts and rediscover useful learning signals from sparse outcome rewards.

Can RL on a small, weaker model improve a stronger student—even when the student already outperforms the small model after RL?

We found that it can—but not by distilling the weak model itself.

Today, we’re excited to share Direct-OPD, joint work w/ @Shiyuan040223, @c7wc7w, @Ahydchh, @Han_lin_Wu, @zhilong_zhang26, Zheng Jiang, @HBX_hbx, Wei-Ying Ma, @yaqinzhang, @haozhou_ai, developed at SIA-Lab @hello_gensi, a joint lab of Tsinghua AIR and ByteDance Seed.

https://t.co/peZTH4QvJi

Our alternative is simple:

1. Run RL on a small model, where exploration and rollouts are cheaper.
2. Treat the model’s pre- and post-RL checkpoints as a teacher pair, whose difference captures the direction learned through RL.
3. On-policy distill this policy shift—what RL changed—using the stronger student’s own rollouts.

In one setting on AIME24:
[*] 1.5B teacher pair: pre-RL and post-RL checkpoints (Post-RL teacher score: 51.3)
[*] 7B student before transfer: 56.7
[*] 7B student + vanilla on-policy distillation: ~50
[*] 7B student + Direct-OPD: 63.1 (+6.4)

The 7B student already starts stronger than the post-RL teacher. Distilling the teacher itself makes the student worse. But distilling what the teacher pair learned through RL improves it further.

In other words, the reusable outcome of an RL run is not only the final checkpoint—it can also be the policy shift encoded by the pre- and post-RL checkpoint pair.