Renato

Lifelong learner. Views my own.

[Renato @Nakarena]

@SawyerMerritt Bummer

[Renato @Nakarena]

@jack Amazing stuff. Congrats!

[Renato @Nakarena]

Part 2: https://t.co/nF9r8ky7wy

[Renato @Nakarena]

@bot Thanks!

[Renato @Nakarena]

https://t.co/JMQiNUirfP

[Renato @Nakarena]

Do you really need to regenerate student rollouts every step of on-policy distillation?

New NVIDIA/UCLA paper: sample once from the initial student, freeze, and keep re-scoring. Across 17 teacher–student pairs it beats OPD on 14, at up to 11.4x lower training cost (partly from shorter rollouts).

The deciding factor is how much the teacher's and student's top tokens overlap at the start (r = −0.73). Low overlap: freeze. High overlap: OPD can still edge ahead.

I made a 10-minute visual walkthrough: intuition first, then the math.

[Renato @Nakarena]

Siyan Zhao, Yonggan Fu and team at NVIDIA and UCLA asked whether on-policy distillation really needs fresh student rollouts every step. Their answer: often no. Sample once, freeze, keep re-scoring. It wins on 14 of 17 teacher-student pairs at up to 11.4x lower cost. I made a visual walkthrough 👇 @siyan_zhao @adityagrover_ @PavloMolchanov

[Yonggan Fu @YongganFu]

Is on-policy distillation really necessary? 🤔

📄 https://t.co/nNLAKjzXhB

🚀 Semi-OPD—distilling on fixed initial-student rollouts—beats OPD on 14/17 teacher–student pairs, with up to +13.6% accuracy and 11.4× training speedup.

More on-policy ≠ better. Alignment matters.

[Renato @Nakarena]

https://t.co/7yYSu1OJCz

[Renato @Nakarena]

Do you really need to regenerate student rollouts every step of on-policy distillation?

New NVIDIA/UCLA paper: sample once from the initial student, freeze, and keep re-scoring. Across 17 teacher–student pairs it beats OPD on 14, at up to 11.4x lower training cost (partly from shorter rollouts).

The deciding factor is how much the teacher's and student's top tokens overlap at the start (r = −0.73). Low overlap: freeze. High overlap: OPD can still edge ahead.

I made a 10-minute visual walkthrough: intuition first, then the math.

[Renato @Nakarena]

Good q. The paper tests it (Table 3, NVFP4). On Qwen3.5-9B MRCR at ≥64k tokens: QUASAR 63.8 vs standard QAT 52.3 (BF16 74.5). So it holds up much better than QAT, but there's still a gap to full precision that grows with context. Long generations also terminate at ~BF16 rates (Table 10).

[Renato @Nakarena]

Your 2-bit model isn't dumb. Your quantizer is.

QAT computes gradients on the rounded weights but updates the full-precision ones, and that mismatch keeps the loss floor high.

QUASAR fixes it at every step using Adam's own statistics, with no extra memory:
��� INT2 Qwen: 34.8 vs 21.5 avg accuracy
• Gemma-4 E4B INT4: 3x lower KL than Google's own QAT checkpoint, at fewer bits
• +1.4% training time, zero inference cost

I made a 7-min visual deep dive, from intuition all the way to the convergence theorem.

@ben_athi @chrismdesa @vcounathe @TianyiZhang_

[Renato @Nakarena]

Want the math without the pain? Part 2 turns every equation into a picture first:

• Quantization as a ruler you slide and stretch
• Curvature-weighted error as a lopsided bowl
• Adam's second moment as a free curvature map
• The closed-form scale/offset fit as a weighted line fit
• The convergence bound as a loss floor you can push down

[Renato @Nakarena]

@Tesla The future looks exciting!

[Renato @Nakarena]

@Tesla 😍

[Renato @Nakarena]

Paper: https://t.co/L8bR7dsM2w

[Renato @Nakarena]

@Tesla Just WOW!

[Tesla @Tesla]

Roadster event livestream https://t.co/G0jU4mWfid

[Renato @Nakarena]

@MacRumors 🤯

[Renato @Nakarena]

@elonmusk Exciting

[Renato @Nakarena]

@DanielStormApps Nice

[Renato @Nakarena]

Same here. Writing code isn't our bottleneck anymore. Reviewing it is. I liked the idea of reviewers checking the intent of a change instead of reading every line, and having approval rules that separate comment-only edits from risky logic changes. Curious how Antique decides what still needs a human.