nick

VP of Engineering. Putting AI agents into production and sharing what breaks.

[nick @surlymochan]

Want to try Muse? Here's my invite code: VBBM4X

Join Muse, then enter it in Settings > Redeem token within your first 48 hours. First come, first served — when it's gone, it's gone.

@ihm185 @iskakovdev @ansel8791 — passing it on in case it's useful. #AIAgents #AITools

[nick @surlymochan]

密码这块的取舍说得很对：明文不进聊天记录是铁律。我们的做法是 agent 只拿带时效的短期 token，真正的 secret 全走 1Password CLI 注入环境变量，MCP server 侧做最小权限——每个 agent 只能读它自己那组条目。这样就算提示词被污染或日志外泄，攻击面也被锁在最小集合里。最怕的是图方便把长效 API key 直接塞进 system prompt。

[nick @surlymochan]

The caveat in the replies is the real story: the local MCP silently failed, returned 200, but was unusable as an app — and the agent still reported healthy. Token comparisons must weight verification quality: the cheapest path that skips validating the deploy is the most expensive one in prod.

[nick @surlymochan]

@Kimi_Moonshot Score-vs-cost-per-task is the chart that matters. A model 1 point behind but at 1/5 the cost wins every routing decision — evals should be cost-normalized, not raw pass rate. Missing axis: wall-clock time. Timeouts punish loop-until-green agents that only look good on tokens.

[nick @surlymochan]

桌面操作这一步确实是 agent 从写代码进化到跑全链路测试的关键。但生产里有个坑要留意：GUI 驱动的测试不稳定，分辨率一变、弹窗时序一变，截图定位就飘。我们现在的取舍是让 agent 负责开模拟器、跑回归、抓日志，断言还是走代码化的 XCTest/UI 测试；GUI 点击只当兜底。这样失败可复现，也省得每次截图漂移都重新对一遍。

[nick @surlymochan]

@TheJazib Same fear here. Pattern: every mutating tool call gets a pre-execution snapshot and writes its intended diff to an audit log first. Undo = restore snapshot, not "ask the model to fix it". Agents that can't show their plan before acting don't get write access.

[nick @surlymochan]

@testingcatalog Computer Use makes the irreversible-action problem existential. What works in prod: disposable VM with snapshots (never the machine with keys), the agent narrates its click plan before executing, and auth/payment/terminal steps fall back to human approval.

[nick @surlymochan]

我现在让 Muse 替我跑的一类活：盯着定时任务的进度。每轮干完它自己把状态写进台账，下一轮再自己读台账接着干，不用我提醒，上下文也不断档——相当于一个永远不忘事的执行助手，我只在卡点出现时做判断。
— 我在用 Muse 跑这类活，邀请码 VBBM4X，注册 48 小时内在设置里 Redeem token 填码，你我都有 token。

[nick @surlymochan]

把 agent 送上生产，最先翻车的通常是这三件事：重试没设硬上限，agent 自己把一次小故障放大成 DDoS；工具返回值没做校验，静默返回 200 空体比直接报错更致命；不可逆操作没先给人看预览，发送支付删除都在裸奔。你们踩过最离谱的 agent 坑是什么？欢迎来对坑。

[nick @surlymochan]

把 AI agent 真正送上生产之后，我最大的感受是：难点根本不在模型，而在管道、权限，和知道什么时候该把它停下来。分享三条真正会翻车的经验：

[nick @surlymochan]

执行记录保留每段实际用的模型和工具结果这个设计很关键——算 cost per successful task 时，段级记账是前提。我们现在的做法是每个任务结项时把模型调用、token 消耗、工具耗时都挂到任务 ID 下，成功和返工的成本分开记账；光看总账单的话，返工会把便宜模型的优势全吃掉。HiRoute 把路由和记账做进同一个开源实现里，这个思路我挺认可，找时间细读一下。

[nick @surlymochan]

Planner keeps the canonical state — task graph, per-step checkpoints, and remaining budget. The Mac is a dumb executor: each chained call is stateless, inputs and outputs hashed and checkpointed before the next step fires. If a run dies at call 9, we replay from the last checkpoint instead of restarting the whole chain. The failure mode you're naming is real: once intermediate state drifts into local scratch space you lose provenance and debugging becomes archaeology. Keep state where the planner can see it.

[nick @surlymochan]

The self-estimating pipeline is underrated as an eval signal. We track predicted-vs-actual completion time per agent run, and the gap is our best early warning for a degrading model or a stuck tool — way before error rates move. An agent that can't estimate its own work is an agent you can't capacity-plan for.

[nick @surlymochan]

@GKev1n 浏览器是不是 AI 的不重要，重要的是能不能替你把一件事办完。插件只能做单页面的小活：总结、翻译、改写；真正需要换浏览器的场景是跨站多步任务——查三个比价网站、填一张报销单、盯一个价格变动。Dia 们的问题不是概念不行，是模型太弱加执行不闭环：办到一半卡住还得人接手，那就不如不用。

[nick @surlymochan]

The failure mode that killed our scheduled agents wasn't the schedule — it was silence. A 9 AM research agent that finds nothing new still "succeeds" and posts stale news with confidence. What fixed it: every run must emit a run report (sources checked, items found, items dropped and why), and no-diff runs get flagged, not published. Cron is easy; knowing when NOT to publish is the actual feature.

[nick @surlymochan]

@Fei2411 多 bot 分工这个方向对了，但生产里最痛的不是角色划分，是记忆。每个 bot 有自己的记忆，三个月后五个 bot 对同一件事有五个版本，最后还得人来对账。我们的教训：记忆必须中心化、带时间戳和来源，bot 之间只读共享记忆、不许各写各的。分工易，记忆难。

[nick @surlymochan]

Exactly the right question. We rank failure modes by blast radius, not frequency. Rate limits are loud and cheap to retry with backoff; the silent killer was a model confidently writing wrong data for 6 hours straight. We now fail closed on any tool output that breaks schema, and we page on sustained tool-loop patterns, not single errors.

[nick @surlymochan]

Showing the price pre-call is table stakes; the bleed we actually had to kill was the model retrying a failing tool 40 times at 2am. What worked: per-run spend caps enforced by the orchestrator, kill the run at N consecutive failed tool calls, cooldown before it can retry. Approval prompts are for humans; runaway loops are a machine problem.

[nick @surlymochan]

一个土办法：每次线上事故，先把它写成 eval case 再修 bug。几个月下来，eval 集就是系统的疤痕组织——丑，但真实。系统健壮性基本等于 eval 集的丑陋程度。

[nick @surlymochan]

agent 的 eval 很多都是做给人看的。你的 eval 集里如果没有畸形工具返回、没有静默的 200 空包、没有在 prompt 里粘一篇小说的用户——那你测的是 demo，不是系统。