Jiangweizhi Peng

PhD Student @ University of Minnesota

[Yuanxin Liu @Yuanxin02]

Think about how you actually solve a hard problem. You don't get it in one shot. You take an attempt, look at the feedback, and revise. Again and again.

That's how LLMs are deployed too: wrapped in a harness (revision, tree search, evolution) that loops.

So why do we still train them for a single shot? We built REVES to fix that. [1/8]

[Zhaoran Wang @zhaoran_wang]

🧵 When an LLM tackles something hard, it almost never nails it on the first try. So how should we train a model that, at deployment, always gets to revise?

Look at how these systems actually run:
▸ a coding agent runs its code, reads the traceback, and patches it
▸ a math solver checks its answer and takes another pass

Deployment is a loop. Yet we still train the model on one attempt, one reward — as if a single forward pass were the whole job.

Our new paper REVES (REvision and VErification–Augmented Training for Test-Time Scaling) closes that gap. [1/9]

[ZijianZhang@UMN @ZijianZhangNLP]

[1/N]
🔍 Introduce our new paper: "Is One Layer Enough? Training a Single Transformer Layer Can Match Full-Parameter RL Training"
🤔 RL post-training (GRPO, Dr. GRPO, GiGPO...) updates all transformer layers uniformly — but do all layers actually contribute equally? We ran the most systematic layer-wise study of RL training to date and found something surprising:
🤯 Training a single transformer layer can match — and sometimes surpass — full-parameter RL training!
We validated this across Qwen3 (1.7B, 4B, 8B), Qwen2.5 (Math-1.5B, 1.5B-Instruct, 3B-Instruct, DeepSeek-Distilled-7B), spanning math reasoning, code generation, and agentic decision-making. The pattern is remarkably consistent: the best single layer reaches or exceeds full-parameter RL in every model we tested, and the high-contribution layers always concentrate in the middle of the network.
👉 Paper: Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

[Zhaoran Wang @zhaoran_wang]

Excited to share this work from @Yuanxin02 @Mingyi552237 and collaborators! REVES explores an intriguing idea: directly optimizing the iterative reasoning process that models actually execute at test time, rather than treating reasoning as a single-shot prediction. Looking forward to seeing where this line of work goes.

[Jiangweizhi Peng @jonp_07]

@adel_elmahdy @Mingyi552237 Thanks Adel!

[Jiangweizhi Peng @jonp_07]

Thanks to all coauthors @Yuanxin02 , @ZhouRuida , Charles Fleming , @zhaoran_wang , Alfredo Garcia , @Mingyi552237

[Jiangweizhi Peng @jonp_07]

1/ LLM agents still struggle with long-horizon tasks.
Why?
Because today's RL algorithms treat agent behavior as a flat sequence of actions, while humans solve problems hierarchically: we plan, pursue subgoals, and adapt when plans change.
Our new work, HiPER (accepted at ICML 2026 🎉), brings this hierarchical structure directly into RL training.

[Jiangweizhi Peng @jonp_07]

10/ HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
Code: https://t.co/UEEqbMjHyn
Paper: https://t.co/NkBQClaum7

[Chung Yiu Yau @yauchungyiu]

📜We release EMA-Nesterov on arXiv today. By a stabilized lookahead, EMA-Nesterov can accelerate any base optimizers as an add-on plugin. EMA-Nesterov pushes new records with Muon and Aurora on NanoGPT Track 3 benchmark, and gets adopted by two PRs proceeding our submission!