Gurpreet

Knowledge speaks wisdom listens. Artificial Intelligence Engineer, Founder -Intencion, #humanist #DeepLearning #Arsenal #Raptors #uWaterlooAlumni 🇨🇦

[Gurpreet @gurpreetgosal_]

@ericweinstein On the bright side now you can assume everything except responsibility.

[Jia-Bin Huang @jbhuang0604]

Incredible open-weight model! The gap between open and closed AI has nearly vanished!

Interested in the tech behind it? Here are some background topics.

Kimi Delta Attention (KDA): https://t.co/V66jlOjfYn

Attention Residual (AttnRes):
https://t.co/2mWgHbiSxv

Mixture of Experts (MoE):
https://t.co/X6GPShwMAw

[Ben Burtenshaw @ben_burtenshaw]

https://t.co/BBZhDdkuxb

[Sean Young @SeanYoung1995]

Alibaba Group and several collaborative research groups have investigated On-Policy Distillation (OPD) in Large Language Models (LLMs) and Vision-Language Models (VLMs), revealing that despite using dense teacher supervision, it induces small and coordinate-sparse parameter updates.

Imagine a sculpture instructor who gently files down a few microscopic rough edges on a student's sculpture rather than rewriting the entire structure from scratch. Checkpoint delta analysis demonstrates that OPD behaves more like sparse on-policy reinforcement learning than dense supervised fine-tuning.

This geometric insight indicates that the on-policy data distribution, rather than reward sparsity alone, acts as the primary driver behind localized and sparse weight updates during model post-training.

https://t.co/lLOfRzO5qp

[Penghui Qi @QPHutu]

PPO is always my favorite RL algorithm, from game to LLM era♥️

DAPO identified a critical issue with PPO’s ratio clipping. However, I don’t think the clip_higher solution addresses the root cause. Our DPPO work (https://t.co/maJpqAOeRr) offers what we believe is a more reasonable solution.

[CyrilXBT @cyrilXBT]

140K GitHub stars. Three months. Most people still copy-pasting into ChatGPT every morning.

Hermes plus Obsidian plus NotebookLM equals a second brain that compounds forever.

Writes its own skills. Maps its own knowledge. Remembers everything you taught it.

One setup. Runs locally. Never starts from zero again.

Save this before you explain yourself to an AI one more time.

Bookmark this now.

[Sasha Rush @srush_nlp]

On-Policy Distillation is the most active new research direction being explored in RL for LLMs. Had the chance to discuss how it works with Dwarkesh and why it fits so nicely into large-scale pipelines.

[Nathan Lambert @natolambert]

Nvidia joined the multi-teacher, on-policy distillation (MODP) gang! Is industry standard post-training right now.

The multi-teacher SFT to RL that Microsoft did in their first model was the standard established by DeepSeek R1. I expect MAI 2 to be MODP.

[Premier League @premierleague]

You did it, @Arsenal 🏆

[Jiaxin Shi @thjashin]

Very interesting work from @zhaoran_wang @YifeiZuoX. Looks like the first working version of higher-order test-time regression extension of softmax attention (cc @heyyalexwang )

[Prime Intellect @PrimeIntellect]

Reward hacking is the hardest problem in RL.

We design settings where hacking is predictable, and find patterns between task difficulty and hack frequency.

These runs are highly efficient, using <$1 in compute. We’re launching Sprints to allow everyone to join this effort.

[Chinmay @ChinmayKak]

One of the papers that really make an interesting contribution to OPSD is this one.

They use consensus gating(do k rollouts from teacher with PI, and the answer that appears the most is the consensus gate; rollouts that agree with the answer are kept, others are discarded)

there are two gates, one at question level, which is if enough rollouts agree on one answer, if not, question is skipped. other is the rollout level gate discussed above.

They primarily derive their gains from teacher generated reasoning, doing full off policy logit distillation on the student conditioned only on the question. They also use a small factor of on policy distillation to get more gains.

Their objective is lambda_off * (masked negative log likelihood) + lambda_on * ( policy-gradient-like weighted log-likelihood)

the on policy part is using clipped advantages, derived from again the log prob ratio similar to other papers. Their method basically prevents leakage to the student via gating, and thus works nicely. Another problem that they also acknowledge is if the model is insufficiently capable, it would agree on wrong answers, thus distilling a noisy signal.

[Ronak Malde @rronak_]

New OPD paper!
On-policy distillation is powerful for merging several distinct RLVR runs of a model without conflict.

But, this is process is currently quite lossy, because of the large distribution shift between the "experts" and the unified model. This paper CoPD proposes that the student and teacher jointly improve, so that their distributions stay close together, but still maintain stable merging.

Really interesting results, and unique approach.

[Yinjie Wang @YinjieW2024]

RLAnything is accepted by ICML 2026~ After one year of research on AI, I finally had first-author papers accepted at NeurIPS, ICLR, and ICML. My theoretical statistics project was also recently accepted by Biometrika. Going forward, I plan to explore more opportunities in industry rather than academia.

[wh @nrehiew_]

https://t.co/p2YYZSijSN

[Perry Dong @perryadong]

Top RL algorithms today are getting powerful — but they can be prohibitively expensive, relying on test-time scaling techniques such as best-of-N sampling

We propose FASTER, a method that maintains the performance gains while eliminating the computational costs

(1/6)

[Walden @walden_yan]

https://t.co/iY0G5ieQhJ

[Ben Eysenbach @ben_eysenbach]

Goal-reaching problems have beautiful mathematical structure, which @princeton_rl and others have leveraged over the last years to design new algs.

Is this useful for general reward-maximization?

Yes! Reward-maximization can be viewed as is just another goal-reaching problem!

[Ifdita Hasan @ifdita_hasan]

Deploying language models in scientific discovery domains requires extraordinary amounts of test-time compute for search algorithms. An ideal training algorithm should be designed with this goal in mind - that we want agents to learn how to not only exploit but also optimistically explore novel strategies. The agent should learn how to synergistically explore and exploit.

We propose Poly-EPO, a set RL algorithm that explores and discovers diverse reasoning paths. Work with @jubayer_hamid (co-lead), Shreya, @ShirleyYXWu, @HengyuanH, @noahdgoodman, @DorsaSadigh, and @chelseabfinn.

[Jeonghye Kim @beanie0__0]

On-Policy Self-Distillation is gaining huge attention as a next-gen LLM post-training method.

It uses the same model as both teacher and student, where the teacher is conditioned on richer information (e.g., solutions). The distributional gap between them enables fine-grained credit assignment on the student's responses, going beyond sparse 0/1 outcome rewards while staying on-policy.

But does self-distillation always work as well as we think?

Making it work on math reasoning seems surprisingly hard 🧐 This GitHub issue, reporting that applying self-distillation to math keeps degrading performance, has been open for 3 weeks with no solid answer yet 👇
🔗 https://t.co/Gf06XRIIx7

🚀 Here is our take on "Why does self-distillation sometimes degrade the reasoning capability of LLMs?"

📌 Key insight: LLM reasoning can benefit from the externalization of uncertainty, especially when dealing with unfamiliar problems. We find that when this is suppressed, performance degrades significantly, particularly when task coverage is broad and OOD settings. In self-distillation, since the teacher is exposed to richer context, the student model trained from it learns to reduce expressions of uncertainty, which in turn leads to performance degradation.

Check out the paper and blog for more details:
Arxiv: https://t.co/gTfYefcNCu
GitHub: https://t.co/eZqBqLeg67
Blog: https://t.co/ldQIoJZ967
Hugging face: https://t.co/CYPGMn4BZE

[Jonas Hübotter @jonashubotter]

Training LLMs with verifiable rewards uses 1bit signal per generated response. This hides why the model failed.

Today, we introduce a simple algorithm that enables the model to learn from any rich feedback!
And then turns it into dense supervision.

(1/n)