Yihao Feng

[Yihao Feng @yihaocs]

congrats!

[Caiming Xiong @CaimingXiong]

Today, we’re excited to launch Recursive (@recursive_si): an exceptional team across London and San Francisco, building AI systems that can safely improve their own capabilities over time.

[Zhihu Frontier @ZhihuFrontier]

🧵 Slime: The Most Elegant & Comfortable RL Training Framework Ever

A deep dive into why Slime redefines LLM RL training with clean architecture & production-grade engineering ✨

Insights from Zhihu contributor Xavier

📌 What Is Slime In One Sentence?
Slime is a streamlined RL training framework built on SGLang (Inference) + Megatron (Training) + Ray (Orchestration).It’s not just a simple stack—it stitches top-tier open-source projects together with perfectly polished interfaces.Core design philosophy: Fully decouple training & inference, connected via streamlined data flow.

Compared to veRL / OpenRLHF:
✅ Native SGLang backend → high concurrency, continuous batching, prefix caching (no messy vLLM wrapper)
✅ Native Megatron backend → full TP/PP/EP/CP parallelism, seamless MoE training
✅ Lightweight Ray scheduling → Placement Group + Remote Actor (no bloated Ray Train)

🏗️ Global Architecture: 3 Modules, One Pipeline
🖥️ Ray Cluster Core Workflow:Data Buffer (Prompt Manager → Buffer & Filter)↔️ Rollout (SGLang → Sampling + RM Scoring + Filtering)↔️ Training (Megatron → Actor/Critic + PPO/GRPO)

🔁 Simplified Core Training Loop
1.Allocate GPU resources via Placement Group
2.Launch SGLang rollout engine
3.Initialize Megatron Actor/Critic models
4.Sync initial weights to SGLang
5.Repeat 3-beat cycle:
Generate (SGLang) → Train (Megatron) → Sync Weights
🎯Elegance = ultra-simple top-level logic, all complexity encapsulated inside modules

🎛️ 4 Core Design Flexibilities
⚙️ Resource Scheduling: Colocate (shared GPU) / Disaggregate (separate GPU pools)
🔄 Training Mode: Synchronous / Asynchronous training
🧪 Sampling Logic: Standard sampling / Over-sampling / Multi-turn tool calling
🤖 Model Type: Dense / MoE, full tensor/pipeline/context parallel support

🔧 Plug & Play Customization (All Extensible)
Slime lets you customize every component via CLI params—no need to fork the repo 🛠️

Key Customization Points
✅ Custom Reward Model: Write an async func to define your own reward logic (easiest entry)
✅ Custom Generate Func: Control multi-turn dialogue, tool calling & external API integration
✅ Custom Rollout Func: Fully take over sampling concurrency & filtering logic
✅ Custom DataSource: Fetch prompts from API / local files / dynamic data streams
✅ Dynamic Filter: Discard low-value sample groups (e.g., zero-variance GRPO samples)
✅ Custom Loss Function: Rewrite PPO/GRPO loss calculation freely
All custom code loads dynamically via --custom-xxx-path config 📝

🚀 Ray GPU Scheduling Magic
Two deployment modes for all cluster scales:
🔹 Colocate Mode: Train & inference share GPUs → high utilization, ideal for small 8-card servers
🔹 Disaggregate Mode: Independent GPU pools → train-infer overlap, perfect for multi-node clusters
Slime stabilizes Ray Placement Group GPU mapping via IP/GPU ID sorting to guarantee reproducibility 🔒

⚡ SGLang Rollout Engine Internals
3-layer abstraction:RolloutManager → RolloutServer → ServerGroup → SGLangEngine
Standout design highlights:
🔸 Over-sampling + Dynamic Filter: Pre-sample extra data, filter invalid groups on the fly
🔸 Async Concurrent Sampling: Process completed groups immediately with FIRST_COMPLETED
🔸 Abort Mechanism: Stop redundant sampling once target data size is met, save compute
🔸 Singleton GenerateState: One-time tokenizer & connection initialization

🧠 Megatron Training Backend
Native support for mainstream RL algorithms:
✅ GRPO: No Critic needed, group-wise reward normalization (most popular)
✅ PPO: Classic Actor-Critic with GAE advantage estimation
✅ REINFORCE++: Token-level baseline optimization
Seamless support for Dense & large MoE models with full parallelism 📊

🔄 Weight Sync: The Hard Engineering Solved
Two high-performance sync paths:
🔹 Colocate: IPC + Gloo → intra-node low-latency weight transfer
🔹 Disaggregate: NCCL Broadcast → cross-node distributed sync
MoE OOM prevention: Chunked Bucket Weight Update → sync parameters in small batches, release memory instantly 🧩

💡 Core Takeaways
✨ Slime’s elegance lies in integrating mature top-tier stacks with clean decoupled design
✨ Minimal top-level logic, maximal internal engineering depth
✨ Fully pluggable customization for all RL scenarios (Math / Code / Agent / MoE)
✨ Optimized for both small single-node & large multi-node clusters

🔗Full article：https://t.co/ig7KCAKCZL
#LLM #RLTraining #SGLang #AIInfrastructure #MoE #MachineLearning

[Recursive @Recursive_SI]

https://t.co/IPHqzO5gWl

[Zhihu Frontier @ZhihuFrontier]

📝 Agentic RL Infra Notes
Insights from Zhihu Contributor 低级炼丹师 📝

🔍 Core Difference: Agentic RL vs Traditional RL
• Traditional RL (RLVR): Single-time generation (answer → reward → update) — trains a "response-generating" model, no dynamic interaction.
• Agentic RL: Continuous action (tools + context + multi-round interaction) 🚀 — trains an "action-executing" model for real-world dynamic tasks.

🧩 Key Challenge: It’s a System Problem, Not Just Long Sequences
🧩 Core pain points: Agent access (white/black-box), environment management, long-tail rollout, training-deployment consistency. 4 systems solve these!

🛠️ Core Goal (Forge): Maximize Training Gain
Formula: Effective Gain = Throughput × Sample Efficiency
Constraints: Support any Agent + Stable convergence.

🌟 Core Solution: Separate Agent from RL Framework
• Agent = Trajectory producer (handles context/tool calls)
• RL System = Collect trajectories + Update models (no Agent simulation!)

🚀 4 Key Systems (1 Sentence Each)
• Forge (MiniMax): 3-layer architecture, supports white/black-box Agents & solves TITO inconsistency.
• ROLL (Alibaba): Splits Agent/environment/training, optimizes rollout bottleneck with Chunked MDP.
• Slime (Zhipu AI): Rollout as HTTP service, fixes TITO mismatch & manages off-policy errors.
• Seer (Moonshot): Sync optimization, splits rollout to cut long-tail latency + model-free speculation.

⏳ Key Optimizations
• Prefix Tree Merging: Cut duplicate computation from shared trajectory prefixes.
• Global KV Cache: Speed up inference for long Agent contexts.
• Clean Environment: Avoid reward pollution from residues/test leaks.

🔧 Deep Dive: Key Technical Points
• Agent Abstraction Layer: Defines unified interface (Observation → Action) to adapt white-box (customizable weights) & black-box (API-only) Agents, ensuring framework compatibility.
• Rollout Optimization: Chunked MDP splits long trajectories into manageable chunks; asynchronous rollout decouples Agent execution from training, reducing latency.
• TITO Consistency: Aligns training (Train)、inference (Infer)、test (Test)、online (Online) environments/Agent versions to avoid performance degradation after deployment.
• Off-Policy Data Management: Uses replay buffer with priority sampling to filter low-quality trajectories, improving sample efficiency; Slime’s HTTP-based rollout ensures data traceability.
• Context Efficiency: Global KV Cache reuses shared context prefixes; Prefix Tree Merging eliminates redundant computation in multi-branch trajectories.

⚠️ Common Pitfalls & Avoidance Tips
• Over-Optimizing Throughput: Ignoring sample efficiency leads to wasted computing resources — balance throughput with priority trajectory sampling.
• Neglecting TITO Mismatch: Training on offline data but deploying to inconsistent online environments causes performance drop — align all four environments (Train/Infer/Test/Online) upfront.
• Agent Over-Simulation: Simulating Agent logic in RL framework increases complexity — stick to decoupling (Agent = trajectory producer, RL = training/collection).

📌 Practical Application Scenarios
• Tool-Using Agents: E-commerce customer service (multi-round tool calls: order query → refund processing) — relies on rollout optimization & context efficiency.
• Autonomous Decision-Making: Industrial control (dynamic adjustment based on real-time data) — benefits from TITO consistency & off-policy data management.

🎯 Epilogue
✅ Agentic RL infra = Decouple Agent + Optimize rollout + Ensure stability
Let’s build better Agentic RL infra together! 🌍

🔗 Highly recommend you to read the full article:
https://t.co/ddysSliDTS
#AgenticRL #AI #ROLL #Agent

[Hayden Prairie @hayden_prairie]

We’ve been thinking a lot about scaling laws, wondering if there is a more effective way to scale FLOPs without increasing parameters.

Turns out the answer is YES – by looping blocks of layers during training. We find that predictable scaling laws exist for layer looping, allowing us to use looping to achieve the quality of a Transformer twice the size.

Our scaling laws suggest that for a fixed parameter budget, data and looping should be increased in tandem!

🧵👇

[Rosinality @rosinality]

The update for the likelihood of correct actions can be very small or even negative, especially in RL with tool use. This might be because of rather OOD tokens from tool outputs and increased uncertainty from it, and structurally shared prefixes among trajectories.

[Rohan Paul @rohanpaul_ai]

The paper shows a way to make LLM reasoning shorter without losing accuracy by rewarding conciseness only when answers are correct.

On a 7B model, it reports 8.1% higher accuracy with 19.9% fewer tokens.

Overthinking is the problem, models make long chains of steps that repeat ideas and burn compute.

Simple length penalties backfire, they either push very short guesses or destabilize training.

So the authors train a small judge that scores a full solution for repetition, relevance, and brevity.

During reinforcement learning, the conciseness bonus applies only if the final answer is correct, which stops reward gaming.

The bonus slowly fades as training goes on, and it shrinks on harder questions so longer reasoning stays allowed.

This cleaner signal reduces gradient noise, makes updates steadier, and avoids the collapse seen with plain length penalties.

Across math tasks and backbones like Qwen, Llama, and Mistral, outputs get shorter while accuracy holds or improves.

----

Paper – arxiv. org/abs/2511.09158

Paper Title: "Efficient Reasoning via Reward Model"

[Rohan Paul @rohanpaul_ai]

New Tencent paper shows how smarter data curation makes reinforcement learning code models much stronger.

The core idea is to stop obsessing over new reinforcement learning tricks and fix the data and training flow.

The team first fine tunes a 32B model on curated coding data and tags each problem by difficulty.

They then run reinforcement learning with real rewards from unit tests that execute the code.

Stage 1 widens the model’s habits by training on a broad mixed set with multiple attempts per prompt.

Stage 2 zooms in on a small pool of the toughest problems and spends many more attempts per prompt.

This 2 stage setup raises pass rates on LiveCode, LeetCode, and Codeforces, with the biggest gains on hard sets.

The ablations show that skipping either stage hurts results, and that easy samples during Stage 2 waste learning.

The largest jump is a 58% relative gain on Codeforces against the same size baseline.

The same recipe also helps a larger mixture of experts model, which means the approach scales.

----

Paper – arxiv. org/abs/2511.06307

Paper Title: "DRIVE: Data Curation Best Practices for Reinforcement Learning with Verifiable Reward in Competitive Code Generation"

[Rohan Paul @rohanpaul_ai]

New Nvidia paper shows how a single LLM can teach itself to reason better.

It creates 3 roles from the same model, a Proposer, a Solver, and a Judge.

The Proposer writes hard but solvable questions that stretch the model.

The Solver answers those questions with clear steps and final results.

The Judge scores both question quality and answer correctness using strict rules.

All 3 roles run in a closed loop and learn with reinforcement learning.

The Proposer’s reward blends quality, difficulty, and clean formatting checks.

The difficulty reward is higher when the Solver fails, which drives harder tasks.

A quality filter blocks weak or confusing questions from entering the training set.

The Solver’s reward comes from the Judge’s score plus a formatting check.

The Judge is trained to output reliable numeric scores inside simple tags.

All role parameters update together using normalized advantages for stability.

On Qwen2.5-3B, the approach lifts average accuracy by 4.54%, and it beats simple supervised fine tuning while rivaling a strong self play baseline without outside tools.

----

Paper – arxiv. org/abs/2510.23595v1

Paper Title: "Multi-Agent Evolve: LLM Self-Improve through Co-evolution"

[vLLM @vllm_project]

🚀 Excited to share our work on batch-invariant inference in vLLM!
Now you can get identical results regardless of batch size with just one flag: VLLM_BATCH_INVARIANT=1
No more subtle differences between bs=1 and bs=N (including prefill!). Let's dive into how we built this 🧵👇

[Yufan Zhuang @yufan_zhuang]

Can LLMs reason beyond context limits? 🤔

Introducing Knowledge Flow, a training-free method that helped gpt-oss-120b & Qwen3-235B achieve 100% on the AIME-25, no tools.

How? like human deliberation, for LLMs.

📝 Blog: https://t.co/9O7TgH5tJs
💻 Code: https://t.co/hHSLzol4Wg

[Rohan Paul @rohanpaul_ai]

The paper shows a few reinforcement learning tweaks let small LLM agents use tools better and beat larger ones.

A 4B model matches or exceeds 32B agents on hard math, science, and code tasks.

Old training stitched fake tool traces together, which taught clumsy timing for tool calls.

They switch to real, end to end tool sessions so the model learns when, why, and how to act.

They also build a diverse, model aware reinforcement set that keeps training exploratory and steady.

Algorithm changes are simple, loosen clipping a bit, add a light length penalty near the limit, and train with token level loss.

This keeps the model’s action entropy in a healthy range so it keeps trying useful options.

On behavior, deliberate agents think a bit longer, call fewer tools, and win more per call.

Reactive agents spam tools and get worse results per call.

Long chain of thought models avoid tools on reasoning tasks, so instruction tuned models scale cleaner for agents.

The recipe is real trajectories plus entropy friendly updates plus deliberate tool use.

This combo turns compact agents into strong problem solvers with efficient tool calls.

----

Paper – arxiv. org/abs/2510.11701

Paper Title: "Demystifying Reinforcement Learning in Agentic Reasoning"

[Simon Yu @simon_ycl]

💥New Paper

Diversity is the key to everything: creative tasks and RL exploration. Yet, most LLMs suffered from mode collapse, always repeating the same answers.

Our new paper introduces Verbalized Sampling, a general method to bypass this and unlock your model's true potential. See the discussion 📷👇

[Ant Ling @AntLingAGI]

2/5
Continuously evolving deep thinking.
Ring-1T scales reinforcement learning from 16B → 100B → trillions of parameters with our icepop algorithm and ASystem.
This ensures stable long-term RL training for large MoE models and bridges the gap between training and inference.
The result: smoother reasoning, longer-context stability, and stronger alignment under trillion-scale load.

[God of Prompt @godofprompt]

🚨 NVIDIA just did the impossible.

They trained a 12B-parameter language model on 10 trillion tokens entirely in 4-bit precision.

It’s called NVFP4, and it might redefine how frontier AI models are trained.

Here’s why this matters:

• NVFP4 delivers 2–3× faster math throughput and 50% less memory vs FP8
• Accuracy? Practically identical. (MMLU-Pro: FP8 = 62.62%, NVFP4 = 62.58%)
• Stability issues? Solved using Random Hadamard transforms, stochastic rounding, and 2D scaling
• Trained entirely on NVIDIA Blackwell GPUs the first 4-bit run stable across 10T tokens

This is the first successful demonstration of large-scale 4-bit pretraining without losing accuracy.

The next generation of frontier models will be faster, cheaper, and greener without compromise.

[Infini-AI-Lab @InfiniAILab]

🤔Can we train RL on LLMs with extremely stale data?

🚀Our latest study says YES!

Stale data can be as informative as on-policy data, unlocking more scalable, efficient asynchronous RL for LLMs.

We introduce M2PO, an off-policy RL algorithm that keeps training stable and performant even when using data stale by 256 model updates.

🔗 Notion Blog: https://t.co/vVUUuTiOL7
📄 Paper: https://t.co/Rmzd9rJHAv
💻 GitHub: https://t.co/xD6i6fRKwl

🧵 1/4

[Dan Alistarh @DAlistarh]

🚀 We are releasing state-of-the-art post-training quantization (PTQ) algorithms for Microscaling FP4, together with kernels:
- First study focused on MXFP4/NVFP4 PTQ for LLMs
- New Micro-Rotated (MR) format and GPTQ algorithm
- QuTLASS GPU kernels with up to 3.6x speedups.

[Zhihu Frontier @ZhihuFrontier]

🚀 @AntLingAGI open-sourced Ling 2.0 — the first FP8-native mixed-precision training framework for MoE models. Plug-and-play. So how does it work? Zhihu contributor & Ling Team dev 千千 shares his thinking.

🧠 Why it matters:
Ling 2.0 is trained natively in FP8, not BF16 with post-hoc quantization. Using fine-grained tile/block-wise scaling, it avoids outlier distortion and retains BF16-level accuracy.

⚙️ Key Results:
• Up to 30-60% throughput gain with MTP
• Still 90-120% faster without MTP
• Training loss curve matches BF16 — virtually no accuracy drop

💡 Innovations include:
• FP8 Optimizer: Compresses 1st & 2nd moments to FP8, saving up to 75% Adam optimizer memory
• On-demand weight transpose: Cuts redundant memory for backward pass
• FP8 routing maps for more efficient MoE dispatching
• Support for E4M3 FP8 format in backward — better accuracy than E5M2

🔍 Tackling FP8 challenges:
• Quantization errors (e.g. outliers clipped to 0)
• Value distortion (close values collapsed)
Ling 2.0 adds monitoring for quantization loss, underflow, and FP32 recompute deltas — alerts when precision degrades.

💭 Why it matters long-term:
In the BF16-vs-FP8 tradeoff, Ling 2.0 emphasizes "lossless precision" first, then optimizes performance. Combined with 3D parallelism and dynamic memory strategies, it's ideal for training large LLMs under memory constraints.

📖 Full technical post:
https://t.co/1EisENr01N
#FP8 #LLM #AIInfra #OpenSource #AntGroup #Ling2

[Zhangchen Xu @zhangchen_xu]

🚀 Want high-quality, realistic, and truly challenging post-training data for the agentic era?

Introducing Toucan-1.5M (https://t.co/0MFs5yIBHZ) — the largest open tool-agentic dataset yet:
✨ 1.53M real agentic trajectories synthesized by 3 models
✨ Diverse, challenging tasks across wide domains
✨ Built from 495 MCP servers & 2K+ open-source tools

📈 SOTA results on BFCL & MCP-Universe with just SFT.
💡 Data, models, and pipeline are all open!

Key Features:
- Multi-tool & multi-server tasks for advanced planning
- Multi-turn interactions with auto-generated follow-ups (super-long trajectories)
- Single + parallel function calls → efficient tool use

[Nouha Dziri @nouhadziri]

🚀Ever wondered how to make RL work on impossible hard tasks where pass@k = 0%? 🤔

In our new work, we share the RL Grokking Recipe: a training recipe that enables LLMs to solve previously unsolvable coding problems! I will be at #CoLM2025 next week so happy to chat about it!

We also dive into the heated debate: does RL just sharpen previous learnt skills or can it unlock genuinely new reasoning? 🔥🔥

Read the full blog here: https://t.co/y8WcPUQELU

#AI #RL #NLP #reinforcementlearning #llm