Sydney He

LLM Researcher at Alibaba | Agentic RL

[Sydney He @helansydney]

@AlexanderLong I’m one of the authors. Feel free to ask me any questions.

[Sydney He @helansydney]

Our team homepage: https://t.co/iy6l52oeOT

[Sydney He @helansydney]

The Bitter Lesson Behind Building Agentic RL in Terminal Environments

This blog post summarizes our practical experience over the past three months working on Agentic RL.

For more details, please refer to: https://t.co/G1OgSlnnwy #LLM #RL #Agent #AgenticRL

[Sydney He @helansydney]

Some Thoughts

[Sydney He @helansydney]

Based on our experience, continuous, fine-grained monitoring and adjustment is critical for maintaining stable and effective long-running agentic RL training (for preventing subtle reward-hacking behaviors).

[Sydney He @helansydney]

Crash Is Normal: How We Made Agentic RL Survive the Terminal

[Sydney He @helansydney]

Tricks we used:
selective token masking
retry-loop penalties
other light behavior shaping rewards or penalties, …
Most of these tricks share the same goal: controlling which trajectories, which parts of trajectories, and with what weight they contribute to policy gradients.

[Sydney He @helansydney]

We propose modeling multi-turn agentic interaction at the level of interaction chunks, typically ending with a tool invocation. Instead of optimizing over individual tokens or entire trajectories, we treat each chunk as a semantic action unit.

[Sydney He @helansydney]

we adopt a simple curriculum: Early on, we update the policy using only positive trajectories, using a large pool of instance data to establish a stable policy manifold. Later, once we have a small set of high-quality instances, we train on both positive and negative trajectories

[Sydney He @helansydney]

Any trajectory that are harmful to training or does not provide valid learning signals can be masked or filtered.

[𝚐𝔪𝟾𝚡𝚡𝟾 @gm8xx8]

Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?

DeltaBench benchmarks LLMs ability to critique long CoT reasoning from o1-like models (e.g., QwQ, DeepSeek-R1) across tasks like math, coding, and general reasoning. It analyzes model effectiveness, evaluates PRMs and critic models for error detection, and identifies limitations in longCoT reasoning.

[Sydney He @helansydney]

@gm8xx8 cool work

[Sydney He @helansydney]

LlamaIndex: How To Evaluate Your RAG (Retrieval Augmented Generation) Applications by @kiennt_ in @BttrProgramming https://t.co/EPnLe0ndzA