Matthew Yang

MSML student @ CMU

[Matthew Yang @_matthewyang]

Almost nobody does proper credit assignment in RL-on-LLMs 💀

Learning only from the final outcome
→ punishes good steps 😭
→ rewards bad steps 😭😭

🚨New Paper🚨
A new paradigm for credit assignment:
LLMs identify their own mistakes ❌ and propose targeted fixes 🎯

🧵[1/n]

[Jack Bai @jackbot_cs]

Tomorrow at ExHall A & F Poster Location: 471, 5pm-7pm, we'll present WebGym, together introducing two recently pre-released works (AsyncWebRL and OpenWebRL) as a surprise :)

WebGym: https://t.co/vUva6lTXah
OpenWebRL: https://t.co/79pJpL1NI3
AsyncWebRL: https://t.co/PdvVJTGoQV

[Boaz Barak @boazbaraktcs]

To be clear, while I believe human mathematicians will have role to play, it does not mean there won't be a dramatic shift, nor that there is not a sense of loss.

I will personally miss the days of being able to sit, with just pen and paper, and discover via pure thought a mathematical truth than no one knew before.

[Ian Wu @ianwu97]

I'll be presenting 3 papers at ICLR 2026! Please stop by for a chat. #ICLR2026 🇧🇷🇧🇷🇧🇷

[idan shenfeld @IdanShenfeld]

I’m flying tomorrow to Brazil for ICLR! 🇧🇷

If you’re into continual learning, self-distillation, learning from textual feedback, or just want to chat—let’s meet!

I'll be presenting the following papers:

[Ian Wu @ianwu97]

1/How can we train LLMs to continually improve their reasoning over test horizons much longer than their training token budgets?

Introducing Reasoning Cache (RC), an algorithm that trains LLMs to *extrapolate*.

[Aviral Kumar @aviral_kumar2]

Can just a 4B model solve IMO-level proof problems at the level of much stronger LLMs like Gemini 3 Pro? Yes, if you can train the LLM to scale test-time compute well!

We're very excited to release our 4B model "QED-Nano", built via an awesome open collab! Details below🧵⬇️

[Matthew Yang @_matthewyang]

@rosmine We tried generating interventions with larger models, namely Qwen3-30B-A3B-Instruct (see section 3) and Gemini 2.5 Pro (see Appendix).

We find that larger models tend to generate better interventions (row 6 vs. row 5).

[Matthew Yang @_matthewyang]

Almost nobody does proper credit assignment in RL-on-LLMs 💀

Learning only from the final outcome
→ punishes good steps 😭
→ rewards bad steps 😭😭

🚨New Paper🚨
A new paradigm for credit assignment:
LLMs identify their own mistakes ❌ and propose targeted fixes 🎯

🧵[1/n]

[Matthew Yang @_matthewyang]

@dhruvbhatia0 No, it does not, because we run standard online RL after we SFT on the interventions

[Aviral Kumar @aviral_kumar2]

🚨🚨New paper

Scaling RL to complex tasks shows credit assignment is a bottleneck

But standard way of fitting PRM + optimizing it is too inefficient to solve it❌

Our idea: use asymmetries in an LLM to let it do its own credit assignment, in natural language w/o PRMs! 🧵⬇️

[Jack Bai @jackbot_cs]

🚨 New Paper Alert 🚨

💥 SFT on hard tasks given reference solution is usually too off-policy, which can cause the training to crash.
🐌 On-policy RL on these hard tasks introduces low sample efficiency, although more stable.
😈 Today, we introduce Intervention Training (InT), an algorithm that avoids shortcomings of both sides.

A thread 🧵 1/n

[Matthew Yang @_matthewyang]

Thank you to my amazing set of collaborators @jackbot_cs @ianwu97 @geneyang4 @setlur_amrith @aviral_kumar2 for making this happen!!! 🙏🙏🙏

And grateful to end my master’s journey with this project ⛵️🌅😎

🧵[7/n]

[Matthew Yang @_matthewyang]

website: https://t.co/cVK44TVQ7n
paper: https://t.co/WTXJfdSsOi
code: https://t.co/wnVaxgLaVB

🧵[6/n]

[Jack Bai @jackbot_cs]

😈 Today, we introduce WebGym, the largest-to-date open-source RL environment for web agent training that contains 300k tasks and a rollout framework optimized specifically for web environments' rollout speed. We reveal the effects of essential scaling directions we observe with WebGym.

1/n

[Aviral Kumar @aviral_kumar2]

🚨🚨New blog post led by CMU students:

Want to know why LLM RL training plateaus on hard problems & scaling compute may not help? And how to fix this issue?

Turns out it stems from a coupling of poor exploration & optimization. Classical ways to explore don't work, but ours does! 🧵⬇️

[Andrew Zhao @_AndrewZhao]

paper of the day