insightLLM

LLM/NLP researcher in Baichuan-Inc @BaichuanAI. Prev @AlibabaGroup, Research Intern at @MSFTResearch
Asia , GitHub:

[insightLLM @insightLLM]

🎉 Introducing our latest work ！
🚀 We propose a label-free method that enables RL without ground truth answer, yet achieves impressive performance on mathematical tasks: 40.0% accuracy on AIME2024 🎯 with a 7B base model.

Paper: https://t.co/ooMFSxYoVf

[Jiayi Weng @Trinkle23897]

Harmony format is finally open-sourced. I still remember 3 years ago (before ChatGPT release) @shengjia_zhao, Daniel and I were brainstorming about the right abstraction for RL training, and that is the start point of the entire harmony library. https://t.co/KDsM0ZMFx7

[insightLLM @insightLLM]

Why does GRPO make responses shorter as training goes on? 🤔

While reproducing Deepseek-R1, we noticed a weird thing: the longer you train with GRPO, the shorter the responses get 📉.Totally opposite to what paper shows(See Figure 1)! Some blame KL divergence, others say the model’s too small (it’s not like we all have the budget to train a 671B LLM 😢).

But we finally found the real culprit💡: length normalization in the formula(See Figure 2 in DeepSeekMath paper). Since the objective maximizes rewards, dividing by |oᵢ| punishes longer outputs. So the model prefers shorter and shorter replies.

And here’s the kicker: Deepseek-R1’s final formula actually removes this normalization!🆘 (See Figure 3)
BTW, both DAPO and Mistral’s new Magistral fixed this too. @dongxi_nlp @chen_mingyang

[insightLLM @insightLLM]

🎉Our latest work!

https://t.co/ooMFSxYoVf

[insightLLM @insightLLM]

@chen_mingyang Thank you for your attention and shares! There have been many similar works recently, and have already been questioned — see https://t.co/W2dJBxOBe4 . @ShashwatGoel7 welcome everyone to discuss our work.

[insightLLM @insightLLM]

Meaningful!💡The reason behind this phenomenon: the base model(from pre-train) already possesses strong mathematical and logical reasoning abilities. The post-train merely serves to elicit these abilities, so the ground truth answers themselves are actually not that important.

[Stella Li ✈️ ICML🇰🇷 @StellaLisy]

🤯 We cracked RLVR with... Random Rewards?!
Training Qwen2.5-Math-7B with our Spurious Rewards improved MATH-500 by:
- Random rewards: +21%
- Incorrect rewards: +25%
- (FYI) Ground-truth rewards: + 28.8%
How could this even work⁉️ Here's why: 🧵
Blogpost: https://t.co/jBPlm7cyhr

[insightLLM @insightLLM]

@StellaLisy ✨We observed that in the early stage of training, the model focuses on learning to produce the correct format. After this brief phase, it can already achieve 85% of the total performance gain.

[insightLLM @insightLLM]

@StellaLisy We’re very excited! In our recent work, we observed a consistent phenomenon: by using GRPO with only a format reward (without any ground truth reward), the Qwen2.5-Math-7B model achieved an accuracy of 72.8% on the MATH500 benchmark.

Paper: https://t.co/ooMFSxYoVf

[Mingyang Chen @chen_mingyang]

🚀 Introducing ReCall, learning to Reason with Tool Call via RL.

- Multi-turn Reinforcement Learning
- No need for supervised data on tool use or reasoning steps
- Empowers LLMs to agentically use and combine arbitrary tools

Fully open-source! A work in progress and we are actively working on it. Stay tuned!
Code: https://t.co/PvA64fV7oX
Blog: https://t.co/B0wTFTZH3U

#LLMs #DeepSeek #reasoning #RL #ReinforcementLearning #ToolUse #FunctionCall #GRPO

[insightLLM @insightLLM]

📚Experiments show that structure constraints and length optimization alone can yield performance comparable to baseline methods that rely on ground-truth answers.

[insightLLM @insightLLM]

🎉 Introducing our latest work ！
🚀 We propose a label-free method that enables RL without ground truth answer, yet achieves impressive performance on mathematical tasks: 40.0% accuracy on AIME2024 🎯 with a 7B base model.

Paper: https://t.co/ooMFSxYoVf

[insightLLM @insightLLM]

✨We find that constructing surrogate signals based on format correctness and response length, combined with the GRPO algorithm, enables effective training of large models to solve mathematical problems.

[insightLLM @insightLLM]

💡Base model is like an excellent student who has already mastered mathematical reasoning skills, but performs poorly on the test paper, it simply needs to develop good answering habits to achieve outstanding results in exams, to unlock the capabilities it already possesses.