Leon

Research engineer, OSS crusader. Working on language models and rl.

prev: null.

rooting for @Jianlin_S

[Leon @ericssunLeon]

@zyh2022 👀

[Tinker @tinkerapi]

Off-policy RL is difficult; runs can suddenly collapse because of async RL staleness or numerical mismatch.

StableReinforce is a new method on Tinker that keeps your RL runs stable, even when learning drifts from sampling.
https://t.co/PfBNb5lxxy

[Leon @ericssunLeon]

the KL estimator only needs the sampled token, log-prob under q and p, which are already available in Tinker's normal importance-sampling interface.

so the fact that StableReinforce requires additional stability info from the sampler means there's more to this. might be top-k logprobs for score centering for example.

[Tinker @tinkerapi]

Off-policy RL is difficult; runs can suddenly collapse because of async RL staleness or numerical mismatch.

StableReinforce is a new method on Tinker that keeps your RL runs stable, even when learning drifts from sampling.
https://t.co/PfBNb5lxxy

[Leon @ericssunLeon]

Now, IPO uses only this distance as its masking criterion.

DPPO on the other hand retains PPO's asymmetric, advantage-dependent clipping. Updates are masked only when they would push the token's probability further away from the rollout policy and the divergence threshold has been exceeded. Updates that move back toward the generator are left untouched.

This means IPO is more conservative, DPPO preserves potentially useful corrective gradients, wheras IPO prioritizes stability at all cost by rejecting highly mismatched tokens altogether.

[Leon @ericssunLeon]

I've been a fan of DPPO for a while. PI has been running it as the default policy-gradient method for a while, but now they're switching to "IPO".

so what's changed?

1/n

[Prime Intellect @PrimeIntellect]

INTELLECT Policy Optimization (IPO)

In async RL, trainer and inference never agree exactly on a token's probability. IcePop masks on the ratio, which misjudges both tails

IPO masks on the probability difference instead

[Leon @ericssunLeon]

DPPO proposes several alternatives, including binary total variation (TV) distance, which measures the absolute probability difference of the sampled token between the current trainer and the rollout policy. Imporance rtio still weighs the gradient, but TV is the masking criterion.

IPO also uses this!

[Leon @ericssunLeon]

@ahatamiz1 @giffmana so given prev arguments, what's your take on east/west discrepancy? a lot of US frontier lab spinoffs almost universally use SWA. if the infra exists and is mature why don't we see greater adoption in the west?

[Leon @ericssunLeon]

@willcb @inductionheads +1

[Leon @ericssunLeon]

so we're really sticking with the window attention huh, nobody wanna have some fun?

[Leon @ericssunLeon]

@ValsAI any quantitative results on the models tendency to peruse such exploits?

[Leon @ericssunLeon]

@htihle any thoughts on the large jumps between just recent model iterations Sol 6 -> Sol 6.1 and Opus 5 -> Opus 5.5?

[Aamir @aaxsh18]

@helloiamleonie sorry to schmidhuber you here but speculative decoding was already explored by noam back in 2018 https://t.co/ZSegJlSBln

[Leon @ericssunLeon]

isn't the SAO objective missing a stop-gradient on f (i.e cispo)?

[Leon @ericssunLeon]

https://t.co/Sm1nnEkZF9

[Leon @ericssunLeon]

https://t.co/Sm1nnEkZF9

[Leon @ericssunLeon]

eagerly awaiting GLM 5.3 weirdml results.

pure rl scaling for a month, with heavy SWE focus, should be very interesting

[Leon @ericssunLeon]

@PandaAshwinee ye just something representative with wall clock would be great i.e in section

> What does 0-mismatch RL cost?

[Leon @ericssunLeon]

wow, incredibly extensive post, and I appreciate how balanced the 0-mismatch cost/benefit discussion is given how much work went into achieving 0.

some thoughts:

- reproducibility deserves more credit as a benefit here. once your optimization signal has mismatch-induced bias baked in, it gets a lot harder to reason about changes to the RL pipeline, algorithmic or infra. did the run move because the algorithm changed, the serving stack changed, or some interaction between the two? reproducibility is kind of foundational to doing good science, and cutting out a major source of hidden variation seems worth it even if it doesn't show up in the headline reward curve
- Async RL shines as an unfortunate but necessary systems tradeoff.
- would've loved to see the comparisons plotted against wall-clock time instead of training steps. 0-mismatch is trading throughput for cleaner signal, and wall-time would make that tradeoff way easier to actually evaluate

[Ashwinee Panda @PandaAshwinee]

We can now RL large MoEs with 0 train-infer mismatch! And doing so can improve performance (pictured task: teach Qwen3.6-35B-A3B to play Wordle). Everything is open-source and we did a bunch of ablations. 🧵

[Leon @ericssunLeon]

@hkproj holy banger 🍿