Qiying Yu

PhD student at Tsinghua AIR @Tsinghua_Uni @AIRTHU1201
Research: DAPO, Emu

[Qiying Yu @qiying_yu]

🚀 Excited to share DAPO!
The current state-of-the-art large-scale LLM reinforcement learning system, which is also open-sourced!

🔥RL performance
Based on the Qwen-2.5-32B pretrained model, DAPO gets 50 points on AIME. This achieves the new SOTA performance using 50% training steps (previous SOTA achieved by DeepSeek R1's GRPO, 47 points on AIME)

🌟Fully open-sourced!
We release all algorithm details, training code, and datasets.

Project page: https://t.co/vfIL46sp3Q
Paper: https://t.co/ykGLnAQI7t
Code & Dataset: https://t.co/ll70qEmakB

Technical details introduced below
#GRPO #DOUBAO #bytedance #LLM #RL #DAPO #Deepseek #Verl

[Fan Zhang @ryanzhangfan]

🚀 Introducing ETT: the 1st end-to-end vision tokenizer tuning approach designed for native multimodal models - jointly optimizing the vision tokenizer and autoregressive LLM for better representation alignment!
✨ Let your tokenizer do more than reconstruct - let it understand!

[Qiying Yu @qiying_yu]

@YouJiacheng @RWKV_AI xs. arxiv ssnr ❌ zhihu ✅

[Qiying Yu @qiying_yu]

@ZeyuanAllenZhu @albe_alfa Thank you for sharing. Respect, and inspired by your journey!

[Qiying Yu @qiying_yu]

#ICLR2025
I am going to present VAPO & DAPO twice at ICLR, two SOTA LLM RL algorithms.

1. The 1-2 pm verl Expo Talk, Apr 26, Peridot 202-203
2. The 3:00-3:30 pm break, Apr 24, at the ByteDance Booth

Welcome and see you there!

[Qiying Yu @qiying_yu]

@tianle_cai Super like to chat :)

[Quentin Gallouédec @QGallouedec]

Overlong filtering has been shown to significantly stabilize learning and improve performance. You can now use it in TRL!

It simply consists in masking the loss of truncated samples.

Principle proposed by @qiying_yu in DAPO, implemented by @shirinyamani 👏

[Qiying Yu @qiying_yu]

@QGallouedec @shirinyamani Thank you for trying our method and we'are happy to hear that it works. That's great!

[Haibin @eric_haibin_lin]

🚀 Introducing VAPO (Value-based augmented PPO), our latest RL method for reasoning models. Trained from Qwen-32B-base model, VAPO achieves 60.4 on AIME 2024, outperforming DeepSeek-zero-32B and DAPO-32B📈.

Built with @verl_project, and yes, we will open source it soon.

Key questions to answer in this paper:
- How to effectively learn the value function in long chain-of-thought (CoT) tasks
- How to address the issue of inconsistent signal density between long and short texts
- How to deal with the sparsity of positive reward signals

Paper 🔗: https://t.co/8reijd93Us
Great work by Yu Yue and team!

[AK @_akhaliq]

China's ByteDance presents VAPO

Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks

present VAPO, Value-based Augmented Proximal Policy Optimization framework for reasoning models. a novel framework tailored for reasoning models within the value-based paradigm. Benchmarked the AIME 2024 dataset, VAPO, built on the Qwen 32B pre-trained model, attains a state-of-the-art score of 60.4

[You Jiacheng @YouJiacheng]

DAPO's clip high (and other insights) really works well. It has been independently reproduced!

Congrats @qiying_yu !

[Qiying Yu @qiying_yu]

@YouJiacheng That's great. Thank you!

[Kyle Corbitt @corbtt]

Lots of good nuggets here. Interestingly, they completely drop the KL divergence penalty and get good results. This mirrors what we're finding in our own experiments. Seems not to be so necessary for RLVR with GRPO. As a bonus, skipping it speeds up training significantly!

[Turing Post @TheTuringPost]

A new RL algorithm!

DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) from @ByteDanceOSS is a fully open-source RL system, that improves training in long Chain-of-Thought (CoT) reasoning.

It achieves 50 points on AIME 2024, surpassing DeepSeek-R1-Zero, using only half the training steps.

What are its key features?

• Clip-Higher – Prevents models from becoming too repetitive and improves reasoning diversity.
• Dynamic Sampling – Makes training faster and more stable.
• Token-Level Policy Gradient Loss – Optimizes learning for long-CoT.
• Overlong Reward Shaping – Reduces noise and improves training stability.

Here are the details:

[Marktechpost AI @Marktechpost]

ByteDance Research Releases DAPO: A Fully Open-Sourced LLM Reinforcement Learning System at Scale

Researchers from ByteDance, Tsinghua University, and the University of Hong Kong recently introduced DAPO (Dynamic Sampling Policy Optimization), an open-source large-scale reinforcement learning system designed for enhancing the reasoning abilities of Large Language Models. The DAPO system seeks to bridge the gap in reproducibility by openly sharing all algorithmic details, training procedures, and datasets. Built upon the verl framework, DAPO includes training codes and a thoroughly prepared dataset called DAPO-Math-17K, specifically designed for mathematical reasoning tasks.

DAPO’s technical foundation includes four core innovations aimed at resolving key challenges in reinforcement learning. The first, “Clip-Higher,” addresses the issue of entropy collapse, a situation where models prematurely settle into limited exploration patterns. By carefully managing the clipping ratio in policy updates, this technique encourages greater diversity in model outputs. “Dynamic Sampling” counters inefficiencies in training by dynamically filtering samples based on their usefulness, thus ensuring a more consistent gradient signal. The “Token-level Policy Gradient Loss” offers a refined loss calculation method, emphasizing token-level rather than sample-level adjustments to better accommodate varying lengths of reasoning sequences. Lastly, “Overlong Reward Shaping” introduces a controlled penalty for excessively long responses, gently guiding models toward concise and efficient reasoning.......

Read full article: https://t.co/9pGFeZw3qh

Project Page: https://t.co/9EZBxD9zyk

[PapersAnon @papers_anon]

DAPO: An Open-Source LLM Reinforcement Learning System at Scale

From a joint ByteDance/Tsinghua team. Proposes the Decoupled Clip and Dynamic sAmpling Policy Optimization algorithm and fully open-sources a SOTA large-scale RL system. Both were used to achieve 50 points on AIME 2024 using a Qwen2.5-32B base model.

Links below

[Maziyar PANAHI @MaziyarPanahi]

To my RL heroes, DAPO!😂

Who’s got insights? Let’s make some magic happen.

🔗 https://t.co/uWcurkWx6K

@willccbb @Dorialexander @UnslothAI @axolotl_ai @huggingface (trl/open-r1)

[Qiying Yu @qiying_yu]

@_philschmid @ByteDanceOSS Thank you for featuring! Excited to sharing insights with the community. 🌟