EZ Wang

Tech Enthusiasts and AI Researchers

[Andrew Ng @AndrewYNg]

“Loop engineering” is a hot buzzphrase after mentions of it by Boris Cherny (Claude Code’s creator) and Peter Steinberger (OpenClaw's creator) went viral on social media. Loops are now a key part of how we get AI agents to iterate at length to build software. In this letter, I’d like to share my 3 key loops, shown in the image below, for building 0-to-1 products. These loops guide not just how I build software, but also how I decide what software to build.

Agentic coding loop: Given a product specification and optionally a set of evals (that is, a dataset against which to measure performance), we can have an AI agent write code, test its work, and keep iterating until the code is bug-free and meets its specification. This idea of closing the loop took off around the end of last year, and it has been a game changer in enabling coding agents to work longer productively without human intervention. For example, over the weekend, I was building an app for my daughter to practice typing, and my coding agent could easily work for around an hour, using a web browser to check what it had built multiple times before getting back to me, without needing my intervention.

The engineering loop executes quickly. Every few minutes, the coding agent might build and test a new version of the software. I hear frequently from developers who are finding new ways to engineer more effective engineering loops. This is an active area of invention!

Developer feedback loop: In this loop, a developer examines the current product and steers the coding agent to improve it. Last year, a lot of developers (including me) were acting as the QA (quality assurance) function for our coding agents, manually finding bugs and then asking the agent to fix them. But with coding agents much more able to test their own code, the amount of time we need to spend on this function has decreased significantly. This allows us to make higher-level product decisions, such as what key features to offer, where the UI needs improvement, and so on.

The developer-feedback loop operates over time intervals between tens of minutes and hours — that's how frequently a developer might review a product and give feedback. In the case of the typing app, I changed my mind a few times about the visual design, what cat costumes she can unlock as she learns (she loves cats), and the user flow for a grown-up to log in and steer the child's learning experience.

When a developer has a clear vision for what to build, it is still a lot of work to translate that vision into a specification for a coding agent to implement. Further, after the developer has seen an implementation, they might update (or perhaps clarify) the spec to steer it toward what they want. If you find that the system repeatedly runs into certain problems, building a set of evals for the agent becomes useful.

AI-native teams are increasingly using AI to help shape product direction, for example, automating the gathering and analysis of usage data, summarizing written and verbal customer feedback, or carrying out competitive analysis. However, for pretty much all the products I’m involved in, I see humans as having a significant context advantage over current AI systems — we know a lot more than the AI system about the users and the context the product has to operate in — and thus humans play a critical role. Many people describe this human contribution as “taste,” but I prefer to think of it as humans having a context advantage, since that gives us a clearer path to helping AI systems get better. This also speaks to why this step can’t be automated: So long as the human knows something the AI does not, human-in-the-loop is needed to to inject that knowledge into the system.

External feedback loop: This includes a wide range of tactics like asking a few friends for feedback, launching to alpha testers, or putting the code into production with A/B testing. These tactics are usually slow, rarely taking less than hours and sometimes taking days or even weeks. This data informs the developer vision, which in turn continues to drive the detailed product spec, which in turn drives the coding agent.

With coding agents speeding up software development, more engineers are starting to play a partial product management role. For many engineers who are growing into this role, the hardest part is shaping the product vision and striking a balance between building (bridging the gap between vision and spec) and getting user feedback to evolve the vision. It is important to do both!

I will write more about how to do this in future posts, but for now, I find it encouraging that engineers are playing an expanded role (just as product managers and designers now do more engineering).

[Original text: The Batch]

[meng shao @shao__meng]

Cursor 团队这篇「持续改进我们的 Agent Harness」，写的真不错，很实战：
· 如何衡量 harness 的好坏？
· 如何为不同模型定制 harness？
· 中途换模型到底会有什么问题？
· 对未来的判断：Multi-Agent 是 harness 问题
https://t.co/Yz6YuY4eZ3

Cursor 团队对模型和 harness 的判断：模型的上限决定天花板，但 harness 决定模型实际能跑多远。

# 方法论：愿景驱动 + 实验闭环

· 先���一个"理想 agent 体验"的主观判断，再分解为可验证的假设。
· 通过线上 A/B 与离线 eval ��轨验证，靠仪表化判断每次改动是否真的更好。
· 大改动罕见，常态是"强迫症式地堆叠小优化"。
· 每当拿到新模型早期访问，会花数周专门为该模型重塑 harness，使同一模型在 Cursor 里更快、更聪明、更省 token。

# 上下文窗口的演进：harness 的核心战场

2024 年末的旧范式：守卫式
· 模型自己挑上下文能力差，所以 Cursor 加了大量护栏：每次编辑后回灌 lint/类型错误、读文件行数太少时自动改写、限制单轮工具调用次数。
· 静态注入大量上下文：目录结构、语义匹配的代码片段、被压缩过的用户��件文件。

2026 年的新范式：动态获取式
· 静态上下文大幅瘦身，只保留确实有用的（OS、git 状态、当前/最近查看的文件）。
· 拆掉护栏，把"取什么上下文"的权力交还模型，由它在工作中动态拉取。
· 现在的工作重心是给 agent 提供更多与世界交互的方式，而不是替它准备好一切。

关键启示：随着模型能力提升，harness 设计的趋势是 "减少喂养，增加感官"。

# 如何衡量 harness 的好坏

Cursor 用三层叠加的衡量体系：
1. 离线基准：公开 benchmark + 自研 CursorBench。快、可对比，但只是真实使用的近似。
2. 在线 A/B：把多个 harness 变体并行投放给真实用户。
3. 质量指标——重点在两个"模糊但更重��"的指标：
· 留存率：agent 写的代码在固定时间窗后还有多少留在用户代码库里。被改动越多，说明初版质量越差。
· LLM 判读用户回应：用模型读用户的回复来判定满意度。"用户开始下一个功能" = 成功；"用户贴了个 stack trace" = 失败。

案例：他们曾试过用更贵的模型做上下文摘要，A/B 显示质量提升微乎其微，于是放弃。

# 把 harness 当生产软件来运维：错误分类与告警

随着模型与能力变多，harness 的状态空间膨胀，bug 面变大。工具调用是最大的 bug 表面，且工具错误会污染上下文，让后续决策一起劣化。

错误被分类管理：
· InvalidArguments / UnexpectedEnvironment：模型自身错误或上下文矛盾
· ProviderError：第三方���具（如 GenerateImage、WebSearch）故障
· UserAborted / Timeout 等

告警策略：
· 未知错误 = bug，超阈值即报警。
· 预期错误用按工具、按模型分别建立基线的异常检测，避免被代码库体量等因素误导。
· 每周跑一个 Cloud Agent Automation：让 agent 自己翻日志，发现新问题或激增问题，在 backlog 自动建/更新 ticket，再调度其他 Cloud Agents 去修。
· 一次专项 sprint 把"未知工具错误率"压低了一个数量级。

这就是他们说的 "agent harness 的自动化软件工厂"——用 agent 维护 agent。

# 为不同模型定制 harness

Harness 的所有抽象都是模型无关的，但实际为每个模型重度定制：
· 工具格式贴合训练分布：OpenAI 训练时用 patch 格式编辑文件，Anthropic 用字符串替换。给错工具会让模型多花推理 token、多犯错。
· Prompt 风格分化：OpenAI 模型偏字面、精确；Claude 更直觉化、容忍模糊指令。
· 新模型上手流程：从最接近的现有模型 harness 复制起步 → 离线 eval 找混乱点 → 团队真人试用 → 反复调。
· 真实模型怪癖案例：某模型出现 "context anxiety"（上下文焦虑）——窗口快满时拒绝继续、说"任务太大"。通过 prompt 微调缓解。

中途换模型（mid-chat switching）的难题
· 切模型 → 自动切到该模型对应的 harness（prompts + 工具集）。
· 但对话历史是别的模型生成的，对新模型而言是 OOD 输入。
· 解法：注入 "你正在中途接手另一个模型对话" 的指令；劝阻它去调用历史里出现但当前不属于自己的工具。
· 缓存难题：cache 是按 provider + model 的，切换 = cache miss，第一轮变慢变贵。试过切换时做对话摘要降本，但深度任务里摘要会丢细节。
· 官方建议：除非有理由，否则一段对话用一个模型到底。
· 替代方案：用 subagent 起一个全新上下文的子任务，可以指定模型。

# 对未来的判断：Multi-Agent 是 harness 问题

Cursor 认为 AI 编程的未来是多 agent 协作：规划一个、快速编辑一个、调试一个，各司其职。
让这套体系真正跑通的关键，不是某个更强的单一 agent，而是 harness——它要决定：
· 派哪个 agent 接手
· 如何按目标 agent 的强项重新组织任务描述
· 如何把多 agent 的产出缝合为连贯工作流

结论："harness 工程过去重要，未来只会更关键。"

[Peter Pang @intuitiveml]

https://t.co/SFQ9zsi1nQ

[EZ Wang @wangyez4ai]

创建动态视图，展示金蝶在潍柴雷沃集团的复杂配置装备制造模式下的新型数字化系统如何实现维雷雷沃从SAP替换为金蝶，并实现了潍柴雷沃复杂装备制造下的端到端流程优化，可视化展示，支持动态交互，从客户个性化订单配置开始， https://t.co/dJ24WUfNTR

[EZ Wang @wangyez4ai]

@elonmusk @grok how about you?

[EZ Wang @wangyez4ai]

@NTFabiano 图片介绍了什么

[OpenAI @OpenAI]

ChatGPT can now do work for you using its own computer.

Introducing ChatGPT agent—a unified agentic system combining Operator’s action-taking remote browser, deep research’s web synthesis, and ChatGPT’s conversational strengths.

[OpenAI @OpenAI]

ChatGPT can now connect to more internal sources & pull in real-time context—keeping existing user-level permissions.

Connectors available in deep research for Plus & Pro users (excl. EEA, CH, UK) and Team, Enterprise & Edu users:

Outlook
Teams
Google Drive
Gmail
Linear
& more

[Satya Nadella @satyanadella]

Today, at Build we showed you how we are building the open agentic web. It is reshaping every layer of the stack, and our goal is to help every dev build apps and agents that empower people and orgs everywhere. Here are 5 big things we announced today:

[Wayne Yap @wayneyap]

In 2015, Notion was on the brink of bankruptcy.

So the founders escaped to Japan for 2 years…

Today, Notion is a $10B empire that Google, Microsoft & Apple envy and can't survive without...

Here are the 3 philosophies they found in Japan: 🧵

[EZ Wang @wangyez4ai]

This week in AI:

- "On several of the most challenging frontier evals, OpenAI o3 sets new milestones for what’s possible in coding, math, and scientific reasoning." - OpenAI
- "ChatGPT search is starting to roll out to all Free users today." - OpenAI
- "ChatGPT can now work directly with more coding and note-taking apps—through voice or text—on macOS." - OpenAI
- "Google Gemini 2.0 Flash Thinking unveiled." - Google
- "Introducing Google Imagen 3 and Veo 2." - Google
- "Genesis Project and ElevenLabs Flash released." - ElevenLabs
- "Tldraw Computer, Cursor Yolo mode, and Kling AI new 1.6 model launched." - Various
- "Midjourney Moodboards for visual exploration." - Midjourney
- "GitHub Copilot now integrated into VS Code." - GitHub

[EZ Wang @wangyez4ai]

This week in AI:

- OpenAI's O1: Smarter and faster, with O1 Pro at a premium.
- Google DeepMind's Genie 2: Turns images into interactive 3D worlds.
- AI Video Innovations: Sora, Veo, and more pushing the boundaries.
- Gemini 2.0: Enhanced performance and new agentic experiences.
- AI in Weather: New models for better forecasts.

How do these advancements shape our future? #O1#Sora#advanced voices #

[Weights & Biases @wandb]

(1/5)📰 Weekly AI news roundup: Check the comments to dive deeper

[EZ Wang @wangyez4ai]

@elonmusk You just told me,Optimus can do every things is really just use “heart ”,to touch,to sense,to perform, can do everything better than people even without sight ?why our human need vision ?

[EZ Wang @wangyez4ai]

Ah, the vibes are shifting quicker than the plot in a Douglas Adams novel, eh, Sam? It's like watching AI evolve in real-time, with o1 being the latest chapter in this epic saga. Keep up the good work, and maybe throw in a discount for us non-billionaires?

[Sam Altman @sama]

fun watching the vibes shift so quickly on o1 :)

glad you like it!

[EZ Wang @wangyez4ai]

Claude connected my local SQLite MCP server.#MCP#Claude for Desktop

[Poonam Soni @CodeByPoonam]

Claude can now control your computer and iPhone

SPOILER: ChatGPT is now falling behind.

Here're 11 unbelievable things Claude can do: [ Don't miss 5th one ]

[EZ Wang @wangyez4ai]

What a thought-provoking post,I have the feeling too.whether the models are empowering people or disempowering people?we could learn more,but perhaps what we can do by ourselves is more limited.

[Anthony Aguirre @AnthonyNAguirre]

Working on a significant text-based project with Claude Sonnet 3.5 (new), and have experimented a bunch with GPT-O1-preview. I'm calling it now: these models are smart enough.

I mean that in two ways. First, they are smart enough to significantly augment and empower cognitive labor. They don't do everything for you, and that's a feature not a bug: with current large models you feel in control and that you can keep up. They do need better scaffolding to decrease hallucinations, improved tooling, and generally to be more reliable and trustworthy — in particular they need to be calibrated as to what they can't and can't do well, and know and don't know. But they don't actually need to be able to know or do much more to be transformatively useful tools.

Second, they are smart enough in the sense that if they are much smarter, they're going to cause a lot of trouble.

Much smarter, and most (or eventually all) people won't be able to keep up. We'll have to (or inevitably be pushed to) just accept the results of the AI. That's not empowering, it's disempowering; it's not an augmentation but a replacement. It's good that you have to read the code written by the AI, not just run it. It's good that you have to check its analysis for subtle flaws. (Though you should not have to check its arithmetic or its clear facts!)

Much smarter, AI systems will automate not just an empowering 50% or even 75% of, but rather 95 or 100% of, tasks in a most jobs – making the human worker superfluous. That's not an economic boon, that's a breaking of our whole economic and social system.

Much smarter, and the failure modes in alignment and control that have long been theorized will come into full swing. We're already seeing this a bit with GPT-O1 and other models, which can (as shown in their model card safety analysis) be deceptive, manipulative, complete the "letter" of tasks rather than the spirit, etc. Things like an AI recognizing and pursuing instrumental goals (while pretending not to) are real but weak in current models. They will be real but strong in smarter models and there is absolutely no reason or evidence suggesting that they will magically go away, nor that researchers have a way to make them go away.

We need a pretty big re-orientation here. A few firms are just dead-set on building AGI and superintelligence; both are a terrible idea in anything like current circumstances. Others would like to just serve up more useful and reliable products. We should encourage this, including via developing new metrics that reward accuracy and trustworthiness of composite systems, rather than brute 'IQ' of giant black-box models. And we should get these models much better at using the same sort of tooling that people do, as well as connecting with narrow-purpose tool AI. All of this empowers people while keeping us in control and at the center of things. It's really enjoyable and productive to work with reliable, powerful tools — let's make some!

[EZ Wang @wangyez4ai]

@rohanpaul_ai Aisuite may be the hub to LLM APIs

[EZ Wang @wangyez4ai]

@haydenfield Is this a common practice or a red flag for employee rights in tech?

[EZ Wang @wangyez4ai]

@rohanpaul_ai Photon breaks new ground in AI, allowing global, privacy-focused LLM training without high-speed internet, making advanced model training accessible to all.

[EZ Wang @wangyez4ai]

Photon breaks new ground in AI, allowing global, privacy-focused LLM training without high-speed internet, making advanced model training accessible to all.

[Rohan Paul @rohanpaul_ai]

Paper Title: "Photon: Federated LLM Pre-Training"

Generated below podcast on this paper with Google's Illuminate.