Lang Feng

PhD student @NTUsg. ByteDance Seed. Scaling RL for LLMs.

[Lang Feng @langfengq]

Open-source "verl-agent" codebase is evolving fast⚡

A scalable, multi-turn reinforcement learning framework for training LLM/VLM-based agents — now with rich features! (see summary in image below🔽)

🚀 Try it out and train your own LLM agents
📎 GitHub: https://t.co/cRGDeGFKP6

[Alexander Panfilov @kotekjedi_ml]

We can finally talk about it:

We found a way to extract hidden reasoning of frontier models using a vulnerability in the APIs of every frontier AI company.

We verified that our reasoning token count matches billed API thinking tokens 1:1 for most of the prompts we queried.

[ethantsliu @ethantsliu]

some distillation techniques:

OPD: log pi - log pi_theta
OPSD: log pi_theta (• | x, y*) - log pi_theta (• | x)
RLSD: A(x, y) x opsd (advantage direction, opsd magnitude)
OPD^2: log π_teacher − log π_teacher-base

[Xiuyu Li @sheriyuo]

SEED turns each completed agent trajectory into a hindsight skill, then re-scores the same sampled actions with and without that skill so a single evolving policy supplies both sparse outcome RL and dense token-level distillation.

SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning
Paper: https://t.co/d0ScMPt3NB

[Lang Feng @langfengq]

worth a read if you’re working on long-horizon agents

[kabi @kakakbibibi]

🚀 The first comprehensive survey on Long-Horizon Agents is released!

📄 Paper: https://t.co/0nm7q39NUg…
⚙️Github: https://t.co/RhaXMwehWM…

[kabi @kakakbibibi]

🥳Seed2.1 Officially Released:
A Next-Generation Agent for Real-World Productivity

📎Blog: https://t.co/FaFidLu1nm
📎Model Card: https://t.co/zutHfVWoKp

[Gorden Sun @Gorden_Sun]

字节发布了Doubao Seed 2.1系列模型，其中的Pro版本，众测时多数任务评分超过Opus 4.6，在我的常用任务里表现非常好，又一个上桌的国模。使用渠道包括豆包桌面版（面向办公场景）、Trae IDE（面向编程场景）、火山引擎（通过API使用）。

[Lang Feng @langfengq]

@forhadsoohan thank you Forhad😄

[Lang Feng @langfengq]

#Seed 2.1 is officially here, bringing stronger AI productivity to real-world scenarios.

It offers more reliable general-purpose agent capabilities, more stable end-to-end coding, and stronger multimodal understanding. Feel free to give it a try.

https://t.co/VEi34cmbgT

[Lang Feng @langfengq]

coming very soon

[Arena.ai @arena]

Seed 2.1 Pro Preview ranks #8 in Code Arena: Frontend, scoring 1539, on par with Opus 4.6. It performs strongly on React apps and lands in the top 10 for five of seven subcategories. In those areas, only a handful of frontier labs rank above it: @Zai_Org's GLM-5.2 and @AnthropicAI's Claude models.

Highlights:
- #7 on the React leaderboard, #14 on HTML
- #6 Brand & Marketing
- #9 Content Creation Tools, Data & Analytics
=#10 Reference Based Design, Consumer Product

This is an early access preview of the model. It will be publicly available in a few weeks.

[Lang Feng @langfengq]

@chelseabfinn @_Alex_Borghi_ Good idea, are there any good ways to handle the long-horizon td learning if there is only an outcome reward?

[Chelsea Finn @chelseabfinn]

Scaling RL to long horizons remains a major challenge.

Long-horizon Q-learning (LQL) prevents compounding bootstrapping errors by bounding the difference in value over long horizons.

It shows large gains over 1-step TD and n-step returns!

Paper: https://t.co/OTk3M6cz8p

[jietang @jietang]

Recent thoughts:
The Shift to Long-Horizon Tasks
The most likely breakthrough this year will be in long-horizon tasks. We are moving toward a stage where Large Language Models (LLMs) learn to complete extended, complex missions by interacting with Agent environments. This is perhaps where the true value of LLMs lies. Take cybersecurity as an example: imagine a model that continuously hunts for software bugs and vulnerabilities. While it sounds like a search process, it’s actually the model learning the high-level intuition and methodology of a professional hacker. Unlike humans, AI can run 24/7 without fatigue. It could potentially find exploits at a much higher frequwill ency and claim bounties on platforms like HackerOne or BugCrowd. It sounds fun, but fundamentally, it's a revolution that displaces the hacker. If even hackers are being "disrupted," one can only imagine the impact on general programmers.

From One-Person to None-Person Companies
Building on long-horizon capabilities, Autonomous Agent Systems (AAS) will inevitably become the next frontier. Last year, we were discussing the rise of the "One Person Company" (OPC). I didn't expect us to move so quickly toward the "None Person Company" (NPC). It’s an ironic twist—we might all end up as NPCs in this new ecosystem.

Engineering the Impossible: Memory and Learning
To realize the vision above, we must solve three technical pillars: Memory, Continual Learning, and Self-Judging.
I used to think these would require massive paradigm shifts and years of research. However, the pressure from both the technical and application sides is so intense that we are seeing these capabilities emerge through ingenious engineering "tricks":
Memory: Long context windows (1M+) and RAG have significantly bridged the gap.
Continual Learning: While true continual learning remains difficult, the release cycles are shrinking. Global models are updated monthly; domestic models are catching up. If we reach weekly updates by next year, it will effectively function as continual learning.
Self-Judging: This remains the most elusive, yet models like Opus 4.7 are already demonstrating early self-correction and judgment capabilities.

The Self-Evolving Endgame
The most difficult—and most promising—path is Self-Evolution. The current wave is incredibly fierce. I suspect that models like Claude may have already achieved a baseline for self-training: writing their own code, cleaning their own data, generating synthetic data, and then training on it. It might "waste" some compute, but it saves the most precious resources: human labor and time. In the LLM era, speed is everything. Rapid iteration is what creates the cognitive gap between leaders and followers. Claude’s rumored 2-million-chip cluster for next year is likely dedicated to exactly this: autonomous model self-training.
Technical Summary:
1M Context: Necessary baseline.
Memory & Continual Learning: Prerequisites, likely solved first via "tricky" engineering.
Harnessing Environments: The breakthrough point.
Self-Judging: The tipping point.
Full Self-Training: The endgame.

Redefining AGI and the Industry
If this is the road to AGI, then AGI’s definition should be the sum of all human collective intelligence, not just an individual’s intelligence. It must possess the creative capacity to produce something as profound as the "Theory of Relativity"—meeting the bar set by Hassabis.
During this transition, every APP will need to be reconstructed as AI-native. In fact, we might move past the concept of APPs entirely. The most significant challenge will be the reconstruction of the operating system itself. In the future, you won’t see a traditional desktop; you will see an LLM OS, where applications are "generated on demand." This challenges the 80-year-old Von Neumann architecture and represents a total upheaval of the computer science industry.

The Irreversible Wave
From completing long-horizon tasks to fully autonomous operations, every sector—Security, Finance, Law, E-commerce—will be reshaped. Many friends have reached out lately, asking how to transform their enterprises to keep pace with AI. But few truly realize that this irreversible process has already begun. As this massive technical wave hits, we must be prepared to act, but we must also start thinking seriously about how to regulate it.

[λux @novasarc01]

post-training/RL details of deepseek-v4:

i) they replace the mixed RL stage used in deepseek v3.2 with on-policy distillation. they still use RL but RL is mainly used to create strong domain specialists first. then they consolidate those specialists into one final model through OPD.

ii) for each domain they start with supervised fine-tuning on high-quality domain-specific data. after that they run RL using GRPO guided by domain-specific prompts and reward signals. the paper does not provide all reward formulas but the important structural point is that each specialist gets its own domain-conditioned reward distribution.

iii) deepseek-V4 supports three explicit reasoning modes: non-think, think high & think max. they trained distinct specialist models under different RL configurations to support different reasoning capacities. for each mode they apply different length penalties and context windows during RL training which changes the model’s output reasoning length.

iv) one of the more interesting parts is for hard-to-verify tasks they say they discard conventional scalar reward models and instead use a Generative Reward Model or GRM. they curate rubric-guided RL data and use a GRM to evaluate policy trajectories. crucially they apply RL optimization directly to the GRM itself and the actor network natively functions as the GRM (that means the same model family is learning both generation and evaluation...i guess this is closer to LLM-as-judge trained with RL...implicitly moving from reward-as-score to reward-as-deliberative-evaluation).

v) deepseek-v4 treats long-horizon tool use as a state-retention problem...instead of flushing reasoning after every user/tool boundary it preserves the full reasoning trace across agentic tool conversations, making context management part of the trained policy interface.

vi) also it collapses auxiliary routing decisions search/no-search, query generation, domain classification, URL reading, etc. into special tokens appended to the main model context...reusing the existing KV cache instead of paying redundant prefill for separate router/controller models.

vii) deepseek-v4 uses full-vocabulary logit distillation. that means instead of only comparing teacher and student on the sampled token they compare the full probability distribution over the vocabulary at each position.

viii) for agentic post-training and evaluation, deepseek builds DeepSeek Elastic Compute (DSec). it is a production-grade sandbox platform with three rust components: a) apiserver b) edge c) watcher...these communicate through custom RPC and scale over deepseek’s 3FS distributed filesystem...the paper says a single DSec cluster manages hundreds of thousands of concurrent sandbox instances...also for each sandbox DSec keeps a globally ordered trajectory log recording every command and result.

ix) the paper does not go in full details of async-rl...it focuses more on fault-tolerant asynchronous execution than on off-policy correction under stale rollouts (also on RL/OPD execution). it does not fully reveal the behavior-policy staleness semantics of the RL/OPD loop.

[Lang Feng @langfengq]

@Junxiao_THU @Ali_TongyiLab thank you, junxiao😊

[Lang Feng @langfengq]

🎉 Thrilled to share that our paper "AgentOCR" (in collaboration with @Ali_TongyiLab) has been accepted to #ACL2026 (Main)!

Tired of exploding memory token costs in long-horizon agent tasks? We alleviate it by turning the agent's history into an IMAGE.

Instead of feeding raw text, AgentOCR renders the history context visually. We let the agent dynamically compress its own "visual memory" using RL, and cache the segments to keep rendering cheap.

Check it out! 👇
Paper: https://t.co/7GbhvTMNtn
Code: https://t.co/6UmFUCqfqq

[Google @Google]

Today, we updated Gemini 3 Deep Think to further accelerate modern science, research and engineering.

With 84.6% on ARC-AGI-2 and a new standard on Humanity’s Last Exam, see how this specialized reasoning mode is advancing research & development 🧵↓

[Lang Feng @langfengq]

@zhuofengli96475 cool😆

[Lang Feng @langfengq]

@us_and_us thank u. surprised you know my snn work, it was a long time ago, lol🤣

[Lang Feng @langfengq]

We are excited to release our paper "Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems"

In this work, we theoretically identify a primary cause of training instability when extending group-based RL to multi-agent systems. We demonstrate that baselines derived directly from GRPO introduce varying degrees of bias across different agent roles, leading to gradient norm instability📉

💡Dr. MAS use a simple yet effective remedy: an agent-wise corrected baseline for advantage estimation. This approach theoretically calibrates gradient scales, enabling stable end-to-end multiagent LLM RL.

Paper: https://t.co/AsWh9G4c9O

[Lang Feng @langfengq]

We are excited to open-source Dr. MAS 🤖⚡️

Dr. MAS is designed for end-to-end RL post-training 🔥 of Multi-Agent LLM systems. It enables agents to collaborate on complex reasoning tasks with:

✨ Flexible agent registry & multi-agent orchestration ✨ Heterogeneous LLMs (shared/non-shared) ✨ Co-training of multiple agents✨ Efficient resource pooling

Code🔗https://t.co/Ptp3mb3ckG

(1/n)

#LLM #Agent #RL #MultiAgent #verl

[Lang Feng @langfengq]

@ai_codedream thank u🥳

[Lang Feng @langfengq]

We are excited to open-source Dr. MAS 🤖⚡️

Dr. MAS is designed for end-to-end RL post-training 🔥 of Multi-Agent LLM systems. It enables agents to collaborate on complex reasoning tasks with:

✨ Flexible agent registry & multi-agent orchestration ✨ Heterogeneous LLMs (shared/non-shared) ✨ Co-training of multiple agents✨ Efficient resource pooling

Code🔗https://t.co/Ptp3mb3ckG

(1/n)

#LLM #Agent #RL #MultiAgent #verl