Ruhan Wang

Computer Engineering PhD @ Indiana University 🎓 | Building self-evolving AI agents | LLM post-training, RL & reasoning

[Ruhan Wang @iu_ruhan]

Open source gives you the code—but not the map.

A seemingly simple change in Codex can quickly turn into a search across 14 implementation sites and 10 files. Miss one, and the change may be incomplete.

Harness Handbook turns the codebase into a human-readable behavior map, showing what the system does, how behaviors connect, and where each one is implemented.

Start from the behavior, follow the map, and inspect only the code that matters.

Want to understand or customize Codex without getting lost in the repository?

https://t.co/0PRZOR0eLo

[Rohan Paul @rohanpaul_ai]

Beautiful paper on agent harness

Agent harnesses fail to evolve cleanly because developers often know what must change, but not where the behavior lives.

A behavior-based code handbook helped coding agents find more required edit sites while using fewer planning tokens.

Agent harnesses control prompts, tools, memory, and execution, but one behavior often spans distant files, functions, and states.

This makes code changes risky because a request describes what should change, while repositories reveal only where code lives.

Harness Handbook reorganizes a repository around runtime behaviors, then links each behavior to verified source locations.

It combines static analysis with LLM descriptions, 3 detail levels, shared-state maps, and automatic updates after edits.

Its guided search starts broad, narrows to likely code, follows call and state links, then checks live source.

Across Codex and Terminus-2 requests, it raised plan win rates by 10.0 and 18.9 points while cutting token use by 12.7% and 8.6%.

– arxiv. org/abs/2607.13285

Title: "Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable"

[elvis @omarsar0]

Great paper on self-improving agent harnesses.

(bookmark it)

If you maintain a production agent harness, finding every file behind one behavior is often harder than writing the edit.

Harness Handbook builds a three-level map from runtime behaviors to source locations using static analysis and LLM-assisted structuring.

Its BGPD workflow guides coding agents from the system overview to relevant stages, functions, and files, then verifies every candidate against current source.

Across 60 modification requests on Codex and Terminus-2, handbook guidance raised planning win rates from 28.3% to 38.3% and from 26.7% to 45.6%.

Planner token use fell 12.7% and 8.6%.

File- and symbol-level F1 improved in all 24 comparisons against GPT-5.5 and Opus 4.8 reference plans. Complete localization misses fell by as much as 25.9 points.

This is a strong pattern for coding agents that need to evolve large harnesses without losing scattered or rarely executed behavior.

Paper: https://t.co/DW7mIEuPr5

Learn to build effective AI agents in our academy: https://t.co/1e8RZKs4uX

[Tencent AI @TencentAI_News]

New paper drop 🔥
As agent harnesses evolve, the hard part isn't the change itself. It's finding where to make the change

🔗https://t.co/wXABY0rMtu 🔗https://t.co/QyzYgpEwcT

[Junyao Yang @TonyworldLove]

Async RL boosts throughput, but staleness causes instability. SAT solves this by adjusting PPO clipping based on policy drift. By measuring log-ratio mismatch, SAT identifies high-staleness tokens and tightens clipping boundaries for updates that push them further off-policy.

[Tong Chen @tomchen0]

Recursive self-improvement and continual learning research drops every day, and people keep asking me what those concepts actually mean.
@TuXinming and I tried to unwrap the popular papers and products a bit: WHAT agents update, and WHEN agents update.

⬇️Blog: https://t.co/U5zkPTtgNz

[yibie @yibie]

推荐腾讯和几所大学刚发的这篇论文。题目很学术但问题是极度实战的：Agent harness 会持续演化——加新能力、适应新 API、换新模型。但每次要改之前，人类开发者或编码 agent 都必须回答同一个问题：这个行为到底在代码的哪些地方被实现了？ 一个大型 harness 里，一个行为可能分散在十几个文件、跨越多个执行阶段、通过共享状态偷偷连接。现有工具按文件和函数组织信息——但修改请求描述的是行为。论文的核心贡献是一个被称为"Harness Handbook"的东西——自动从代码库生成"行为→代码"的映射，然后让编码 agent 按行为出发来找代码，而不是按文件名遍历。实验效果：更准、更快、用更少 token。

Harness Handbook：让演化的 Agent Harness 可读、可导航、可编辑

腾讯 HY LLM Frontier + Indiana University + University of Maryland 等，2026 年 7 月 14 日。

问题：行为定位是 harness 演化的中心瓶颈

现代 AI agent 的能力不只取决于基础模型，还取决于它的 harness——构建 prompt、管理状态、调用工具、协调执行的那层代码。随着模型、API、执行环境和应用需求变化，harness 必须持续修改。

在人类开发者或编码 agent 能做出修改之前，他们必须识别出所有实现目标行为的代码位置。 这就是"行为定位"——而它很难，因为在生产级 harness 中，一个行为往往分散在多个文件、多个函数、多个执行阶段和状态转换中。

现有的方法——仓库地图、代码搜索、代码摘要、长上下文编辑——按文件和函数组织信息。但它们不告诉你这些碎片如何协同产生行为，也不告诉你是否找到了每一个受影响的位置。编码 agent 必须自己在这些碎片之间建立连接。 这个缺失的连接就是这篇论文要解决的问题。

解决方案：Harness Handbook

不是按文件和函数组织实现知识——而是按"系统做什么行为"来组织，并把每个行为链接到对应的源码。

自动化生成

Handbook 实例通过以下步骤自动从代码库生成：

1. 静态程序分析——提取函数调用图、控制流、数据流
2. LLM 辅助行为结构化——将分析结果转化为行为描述，组织成"行为→代码"的映射

行为导向渐进式揭示（BGPD）

有了 Handbook 之后，编码 agent 的修改流程变成：

1. 从高层行为描述出发——"我要改的是这个行为"（而不是"我要遍历哪些文件"）
2. Handbook 指向相关代码位置——按行为，不是按文件名
3. 验证候选位置与当前源码的一致性——Handbook 是自动生成的，需要验证是否仍然准确
4. 规划具体的编辑——现在知道了"在哪改"

实验

在两个开源 agent harness 上测试了多种修改请求。比较了"有 Handbook 辅助的规划"和"没有 Handbook 的规划"。

结果：
• 行为定位更准确——找到了更多相关的代码位置
• 编辑计划质量更高——计划更完整，遗漏更少
• 用更少的 planner token——不需要在代码库里反复搜

收益最大的场景：
• 涉及分散实现点的修改（一个行为在多个文件里）
• 涉及罕见执行路径的修改（只在特定条件下触发的代码）
• 涉及跨模块交互的修改（行为跨越不同子系统）

为什么这重要

这篇论文做了一个很明确但之前没人系统化做的事：把"行为定位"定义为一个独立的问题，并给出了一个自动化的解决方案。

对于我们的读者来说，可以直接应用的洞察：

1. 写 harness 时，行为文档应该是自动生成的，不是手写的。 手写文档在第一次修改后就过时了。Handbook 的核心思想是：从代码中提取结构，用 LLM 生成人类可读的表达。
2. 编码 agent 的瓶颈不是"写代码"，而是"知道在哪写代码"。 如果你让 agent 去改一个大型 harness，最费 token 的环节是它遍历文件试图找到正确的修改位置。把行为映射提前建好，规划效率会大幅提升。
3. 行为定位收益最大的是"最难找"的代码——分散的、罕见的、跨模块的。 这恰好是人类开发者也最头疼的场景。

原文：https://t.co/JLh4HS7DJL
项目链接：https://t.co/z9K26OClTY

#Agent工程 #Harness设计 #论文

[Ruhan Wang @iu_ruhan]

@_akhaliq Other Related Blog:
- https://t.co/7srFpMWKnB
- https://t.co/H7Hh2N2PfL

[Ruhan Wang @iu_ruhan]

Big thanks to @_akhaliq for sharing Harness Handbook 🙌
Really happy to see the project reaching more people and sparking more discussion around agent harnesses.

- Blog: https://t.co/0PRZOR0eLo
- GitHub: https://t.co/3IpVrt6MTq
- Codex Handbook: https://t.co/e47HFkUCCT
- Terminus-2 Handbook: https://t.co/kCHSYohkD6
- Studio Demo: https://t.co/nECJtD8Y6O
- Paper: https://t.co/8EJGnIgJmS

[Ruhan Wang @iu_ruhan]

@_akhaliq Github Code: https://t.co/VqeWK4Xnny

[DailyPapers @HuggingPapers]

Harness Handbook

Turn any complex agent codebase into a navigable behavior map. Organize by what the system does, not by file structure — with every behavior linked to its code evidence.

[AK @_akhaliq]

Harness Handbook

Making Evolving Agent Harnesses Readable,Navigable, and Editable

[Elon Musk @elonmusk]

Grok 4.5 reaches #1 position on Long-Horizon Terminal-Bench

[Rahul @sairahul1]

This is the best site on the internet to learn harness engineering.

Free. Completely.

Most AI engineers have never heard the term.

https://t.co/bwDbTTYsjM

Bookmark this site.

Then read this setup ↓

[Lilian Weng @lilianweng]

new post on harness engineering for AI self-improvement: https://t.co/ZYvGfVs61k

It is hard to forecast how much the future of RSI will rely on harnesses. Likely harness engineering will evolve in the direction of self-improvement and enable auto-research, and, in turn, smarter models keeps harnesses simple.

Even when many harness improvement get eventually internalized into core model, the need to specify goals and context will not disappear.

[Yucheng Shi @Yucheng__Shi]

Most agent benchmarks end in minutes. Real terminal work doesn’t.

We’re introducing Long-Horizon Terminal-Bench (LHTB), a benchmark designed to measure whether AI agents can sustain progress across hundreds of dependent actions, not just start a task and solve the easy first steps.

LHTB includes:

→ 46 reproducible terminal tasks across 9 categories
→ 18 frontier models evaluated under the same Terminus-2 harness
→ Up to 90 minutes and ~120–320 agent steps per task
→ Hidden, replay-based verifiers that resist shortcuts and reward hacking
→ Continuous partial credit, so meaningful progress isn’t flattened into a binary zero

What did we find?

• The best mean reward is only 0.505
• No model solves even one-third of the benchmark
• 29 of 46 tasks have never been solved by any model
• Higher inference cost does not necessarily mean better performance

Today’s agents can often begin difficult workflows. The harder question is whether they can preserve state, recover from dead ends, verify their work, and actually finish.

Long-horizon execution remains a wide-open frontier.

Watch the video and explore the paper, code, results, sample runs, and leaderboard:

https://t.co/LhCvYrLP0E

[Ruhan Wang @iu_ruhan]

@Xrey_Trading @Yucheng__Shi That’s pretty cool — having the whole flow tracked with transaction IDs would make debugging way less painful.

[Ruhan Wang @iu_ruhan]

@aiseomastery @Yucheng__Shi Yeah! the behavior gives you the big picture, and the code shows you how it actually happens.

[Ruhan Wang @iu_ruhan]

@nickventuri @Yucheng__Shi Haha, true 😄 We’re trying to make that “approve” come with more confidence.