Jiafan He

CS Ph.D. Candidate @UCLA｜@amazon fellow | top reviewer for @NeurIPS 2023

[Jiafan He @JiafanHe]

@Zhiyong16403503 Thank you

[Jiafan He @JiafanHe]

Thrilled to be attending #NeurIPS2024! 🎉 This year, I am on the academic job market and will be presenting my latest research on RL and RLHF at both the main conference and workshops.
Excited to engage in insightful discussions with the incredible community here! Let's connect—feel free to direct message or email me, or swing by the posters. Looking forward to meeting and collaborating with you all! 🙌 #RL #RLHF #ML #JobMarket

[Jiafan He @JiafanHe]

🗓️ Schedule update: Poster session delayed to this afternoon! 😅 Stay tuned...

[Jiafan He @JiafanHe]

RT @JiafanHe: Thrilled to be attending #NeurIPS2024! 🎉 This year, I am on the academic job market and will be presenting my latest research…

[Quanquan Gu @QuanquanGu]

Today’s the day to launch! Introducing MARS (Make vAriance Reduction Shine): the ultimate LLM optimizer. Let’s unite, innovate, and take our shot at MARS! 🚀🚀🚀

Paper: https://t.co/kRjfCMFoCU
Code: https://t.co/HZMz3dZkUV

[Rohan Paul @rohanpaul_ai]

Can RLHF be accelerated by momentum? This paper answers this question in the affirmative.

Accelerated Preference Optimization (APO) accelerates LLM alignment using momentum, improving RLHF performance and convergence speed.

applies Nesterov's momentum to preference optimization

**Original Problem** 🔍:

RLHF has become crucial for aligning LLMs with human preferences. Direct Preference Optimization (DPO) simplifies this process but lacks acceleration techniques.

-----

**Solution in this Paper** 🛠️:

• Applies Nesterov's momentum to iterative preference optimization
• Unifies existing preference optimization algorithms
• Incorporates extrapolation step after each policy update

-----

**Results** 📊:

• APO with 3 iterations achieves 31.73% length-controlled win rate on AlpacaEval 2.0
• 1.78% improvement over iterative DPO, 5.34% over Snorkel's Mistral-PairRM-DPO
• APO with 2 iterations matches 3-iteration iterative DPO performance
• MT-Bench average score: 9.57 out of 10 for general instruction-following tasks

------

Generated this podcast with Google's Illuminate.

[Jiafan He @JiafanHe]

🔗 Update Paper Link: https://t.co/FANQ08ACSA

[Jiafan He @JiafanHe]

(1/n)🚀 Introducing Accelerated Preference Optimization (APO) for Reinforcement Learning from Human Feedback (RLHF)! 🎯

RLHF plays a central role in aligning large language models (LLMs) with human preferences. Direct Preference Optimization (DPO), one of the most popular approaches, simplifies this process by avoiding explicit reward estimation. But what if we could make it even faster and more efficient? The answer: momentum! ⚡

💡 APO Framework: We introduce Accelerated Preference Optimization (APO), a general framework that unifies several preference optimization methods and incorporates Nesterov's momentum to accelerate the convergence process. By framing preference optimization as a proximal point method, we demonstrate APO's ability to speed up the RLHF process.

📊 Theoretical Insights: APO achieves a faster convergence rate compared to standard methods like iterative DPO and Self-Play Preference Optimization (SPPO), which a significant improvement in the efficiency of training LLMs.

🚀 Empirical Results: On the AlpacaEval 2.0 benchmark, APO outperforms DPO, iterative DPO, and other strong baselines in aligning LLMs with human preferences.

🔗 Paper: https://t.co/jUnIw0UW6Z

[Jiafan He @JiafanHe]

(11/n) A joint work with: @HuizhuoY and @QuanquanGu

[Jiafan He @JiafanHe]

(10/n) Theoretical Insights: With an additional assumption on the sub-optimality gap, we prove that the optimized policy in APO converges to the optimal policy at an exponential rate on the Bradley-Terry (BT) model. Furthermore, the introduction of Nesterov’s momentum continues to accelerate the convergence rate.