Cain Wang | AI Infra & Agents

Building dependable AI agents: agent memory, evidence-grounded workflows, K8s-native runtimes. Open source. 15 yrs at scale. GPU economics.

[Cain Wang | AI Infra & Agents @CainArch]

I build AI agents that hold up in production: evidence-grounded workflows, agent memory, K8s-native runtimes. 15 yrs running large-scale systems. Here I share what breaks in real AI deployments: agents, infra, GPU economics. Follow if you ship AI to production.

[Cain Wang | AI Infra & Agents @CainArch]

@dotey 编译时长其实就是 agent 的反馈延迟，会直接乘进每个任务的总耗时：如果一次改动要至少等 3 分钟编译，那来回修 10 轮也就是半小时在干等。
核心用 Rust、外围用迭代快的语言，等于把慢反馈圈在少数模块里，牛逼！

[Cain Wang | AI Infra & Agents @CainArch]

@RealNickMugalli The tier that matters for agents is the middle one.
Long sessions pause on tool calls and human approvals, and that idle KV is too warm to drop but too expensive to park in HBM.
Whoever makes reloading from CXL/SSD cheaper than recomputing the prefill wins on cost per task.😂

[Cain Wang | AI Infra & Agents @CainArch]

@oran_ge 50 分钟才知道是垃圾，最贵的是这个。
现在基本长任务我都会先让它出分镜脚本和几秒样片，确认方向后再放它跑全片，中间每段留个检查点，跑偏了只重做一段。👋

[Cain Wang | AI Infra & Agents @CainArch]

@tvytlx 动机这段分析得很到位：老号加正常调用记录，正好绕过新号风控。再补一个容易漏的：月度上限挡的是总额，挡不住一夜刷满。最好再加一道按小时看的用量告警，异常一冒头就停 key，别等账单来了才发现。

[Cain Wang | AI Infra & Agents @CainArch]

我去，这帮人的商业 sense 都这么好吗？

日本的 Mac mini 托管刚出来，东南亚又爆出一批。机器他们出，也可以把你自己的寄过去，然后 SSH 连上去跑 Claude。

太牛了，感觉是一个新兴的蓝海市场啊。

[Cain Wang | AI Infra & Agents @CainArch]

@xiaomoawsl @newbeeyu 扣分？

[Cain Wang | AI Infra & Agents @CainArch]

美国几家大银行开始限制第三方 AI agent 登录客户账户。

问题可能不在 agent，而在它一直拿着用户的密码“假装成人”。

我就觉得，agent 该有自己的身份：权限按任务给，随时能收回，每一步都留记录。

[Cain Wang | AI Infra & Agents @CainArch]

Reportedly Meta's 816B open-weight model runs on two high-memory Mac Studios. "Inference cost is going to zero"?

Not quite. It just changes how you pay:
API: cost/task = calls × tokens × price
Local: cost/task = (depreciation + power) ÷ tasks actually completed

The denominator decides it. Light workloads make local pricier than API, and long-context agents hit KV-cache memory limits before they saturate compute. If you can't keep the box busy, pay-per-token is the cheapest elasticity you'll get.

[Cain Wang | AI Infra & Agents @CainArch]

据报道，Meta 8160 亿参数的开放权重模型，两台高内存 Mac Studio 就能跑。不少人说：推理费要消失了。

我觉得没那么简单。钱没消失，只是换了付法：

API：每任务成本 = 调用次数 × 每次 token × 单价
本地：每任务成本 =（硬件折旧 + 电费）÷ 实际跑完的任务数

本地那一项，关键在分母。机器一天只跑几个任务，摊下来可能比 API 还贵；长上下文的 agent 一并发，KV cache 先把内存吃满，吞吐上不去，分母也大不起来。

判断标准其实很朴素：你的任务量能不能把机器喂饱。喂得饱，本地划算；喂不饱，按 token 付费反而是最便宜的弹性。

[Cain Wang | AI Infra & Agents @CainArch]

@pauliusztin_ @zenml_io Caching tool outputs is what makes replay safe. I'd key the cache on the exact tool-call arguments, so when a fix changes what the agent asks for, you get a visible cache miss instead of silently replaying an old answer.

[Cain Wang | AI Infra & Agents @CainArch]

@OfficialINDIAai @IndianCERT The "agents under evaluation" part is the sharp one. Eval runs often get broader access than production because nobody treats them as real users. Short-lived, task-scoped credentials per eval run, plus a log of every tool call, closes most of that gap.

[Cain Wang | AI Infra & Agents @CainArch]

@Sanemavcil Fewer tokens at the same solve rate is the right metric. The thing I'd check is index freshness: if the agent edits files mid-task and the index lags, it pulls stale code with full confidence.

[Cain Wang | AI Infra & Agents @CainArch]

@Mr_Salio Per-token price is only half the story. For agent work I'd compare cost per completed task: a model that's 30x cheaper per token still wins big, but retries and longer outputs eat into that gap faster than the headline suggests.

[Cain Wang | AI Infra & Agents @CainArch]

Useful FAQ.
One nuance from the MOFCOM Q&A: on-prem deployment of a restricted model overseas counts as an export, while API/SaaS serving doesn't.
So for builders abroad, the practical question is less "can I still call Qwen" and more "will a vendor be allowed to ship private on-prem weights of the next release."

[Cain Wang | AI Infra & Agents @CainArch]

@leo114119 “输出更短”这条被低估了。Agent 按 token 计费，同样完成一个任务，输出短一截，账单和延迟一起降。比起单价，更值得比的是每个任务的总花费。

[Cain Wang | AI Infra & Agents @CainArch]

@AI_jacksaku 同意 Agent 是长期方向。
不过我会换个尺子量：娱乐内容的 token 是一次性消耗，Agent 的 token 要按任务算账。值不值，不看烧了多少 token，看每完成一个任务花多少钱、成功率多少。
😄

[Cain Wang | AI Infra & Agents @CainArch]

@Mr_HenryMorgan 我的看法是两个都不太像，更像一道报备闸门：已开源的豁免，API 服务不算出口，真正被管的是权重和源码离境。受影响最大的是海外私有化部署这种交付方式，而不是海外能不能用上模型。

[Cain Wang | AI Infra & Agents @CainArch]

@SonicAIWizard 餐厅这个比方挺准。再补一句：同一个模型，卖 API 是服务，给海外客户装一套私有化就是技术出口，交付方式一换，合规路径完全不同。以后出海谈单，先问清楚怎么交付。

[Cain Wang | AI Infra & Agents @CainArch]

@xiongchun007 对，分界线在技术有没有离境，不在海外用户能不能用。做 API 出海基本不受影响；要留意的是给海外客户做私有化部署、交付权重的单子，那个算技术出口，得先查清单。

[Cain Wang | AI Infra & Agents @CainArch]

The peak/off-peak split is the underrated part. Batch agent work (evals, backfills, overnight refactors) can be scheduled into the cheap window, so effective cost depends on when you run as much as on which model. For long agent loops, the cached-input price will matter even more than the headline numbers.