Cain Wang | AI Infra & Agents

Building dependable AI agents: agent memory, evidence-grounded workflows, K8s-native runtimes. Open source. 15 yrs at scale. GPU economics.

[Cain Wang | AI Infra & Agents @CainArch]

I build AI agents that hold up in production: evidence-grounded workflows, agent memory, K8s-native runtimes. 15 yrs running large-scale systems. Here I share what breaks in real AI deployments: agents, infra, GPU economics. Follow if you ship AI to production.

[Cain Wang | AI Infra & Agents @CainArch]

@xiaomoawsl @newbeeyu 扣分？

[Cain Wang | AI Infra & Agents @CainArch]

美国几家大银行开始限制第三方 AI agent 登录客户账户。

问题可能不在 agent，而在它一直拿着用户的密码“假装成人”。

我就觉得，agent 该有自己的身份：权限按任务给，随时能收回，每一步都留记录。

[Cain Wang | AI Infra & Agents @CainArch]

Reportedly Meta's 816B open-weight model runs on two high-memory Mac Studios. "Inference cost is going to zero"?

Not quite. It just changes how you pay:
API: cost/task = calls × tokens × price
Local: cost/task = (depreciation + power) ÷ tasks actually completed

The denominator decides it. Light workloads make local pricier than API, and long-context agents hit KV-cache memory limits before they saturate compute. If you can't keep the box busy, pay-per-token is the cheapest elasticity you'll get.

[Cain Wang | AI Infra & Agents @CainArch]

据报道，Meta 8160 亿参数的开放权重模型，两台高内存 Mac Studio 就能跑。不少人说：推理费要消失了。

我觉得没那么简单。钱没消失，只是换了付法：

API：每任务成本 = 调用次数 × 每次 token × 单价
本地：每任务成本 =（硬件折旧 + 电费）÷ 实际跑完的任务数

本地那一项，关键在分母。机器一天只跑几个任务，摊下来可能比 API 还贵；长上下文的 agent 一并发，KV cache 先把内存吃满，吞吐上不去，分母也大不起来。

判断标准其实很朴素：你的任务量能不能把机器喂饱。喂得饱，本地划算；喂不饱，按 token 付费反而是最便宜的弹性。

[Cain Wang | AI Infra & Agents @CainArch]

@pauliusztin_ @zenml_io Caching tool outputs is what makes replay safe. I'd key the cache on the exact tool-call arguments, so when a fix changes what the agent asks for, you get a visible cache miss instead of silently replaying an old answer.

[Cain Wang | AI Infra & Agents @CainArch]

@OfficialINDIAai @IndianCERT The "agents under evaluation" part is the sharp one. Eval runs often get broader access than production because nobody treats them as real users. Short-lived, task-scoped credentials per eval run, plus a log of every tool call, closes most of that gap.

[Cain Wang | AI Infra & Agents @CainArch]

@Sanemavcil Fewer tokens at the same solve rate is the right metric. The thing I'd check is index freshness: if the agent edits files mid-task and the index lags, it pulls stale code with full confidence.

[Cain Wang | AI Infra & Agents @CainArch]

@Mr_Salio Per-token price is only half the story. For agent work I'd compare cost per completed task: a model that's 30x cheaper per token still wins big, but retries and longer outputs eat into that gap faster than the headline suggests.

[Cain Wang | AI Infra & Agents @CainArch]

Useful FAQ.
One nuance from the MOFCOM Q&A: on-prem deployment of a restricted model overseas counts as an export, while API/SaaS serving doesn't.
So for builders abroad, the practical question is less "can I still call Qwen" and more "will a vendor be allowed to ship private on-prem weights of the next release."

[Cain Wang | AI Infra & Agents @CainArch]

@leo114119 “输出更短”这条被低估了。Agent 按 token 计费，同样完成一个任务，输出短一截，账单和延迟一起降。比起单价，更值得比的是每个任务的总花费。

[Cain Wang | AI Infra & Agents @CainArch]

@AI_jacksaku 同意 Agent 是长期方向。
不过我会换个尺子量：娱乐内容的 token 是一次性消耗，Agent 的 token 要按任务算账。值不值，不看烧了多少 token，看每完成一个任务花多少钱、成功率多少。
😄

[Cain Wang | AI Infra & Agents @CainArch]

@Mr_HenryMorgan 我的看法是两个都不太像，更像一道报备闸门：已开源的豁免，API 服务不算出口，真正被管的是权重和源码离境。受影响最大的是海外私有化部署这种交付方式，而不是海外能不能用上模型。

[Cain Wang | AI Infra & Agents @CainArch]

@SonicAIWizard 餐厅这个比方挺准。再补一句：同一个模型，卖 API 是服务，给海外客户装一套私有化就是技术出口，交付方式一换，合规路径完全不同。以后出海谈单，先问清楚怎么交付。

[Cain Wang | AI Infra & Agents @CainArch]

@xiongchun007 对，分界线在技术有没有离境，不在海外用户能不能用。做 API 出海基本不受影响；要留意的是给海外客户做私有化部署、交付权重的单子，那个算技术出口，得先查清单。

[Cain Wang | AI Infra & Agents @CainArch]

The peak/off-peak split is the underrated part. Batch agent work (evals, backfills, overnight refactors) can be scheduled into the cheap window, so effective cost depends on when you run as much as on which model. For long agent loops, the cached-input price will matter even more than the headline numbers.

[Cain Wang | AI Infra & Agents @CainArch]

@liyue_ai 每个 token 的 KV 缓存压到 890 字节，这个数对多轮 Agent 比跑分更关键：一张卡能扛多少并发会话，基本跟它成反比。我会盯一下压缩后长对话后段的召回，别省了显存，换来第 30 轮忘了第 3 轮的约束。

[Cain Wang | AI Infra & Agents @CainArch]

@RealYDT 原文里最容易被忽略的是登记这一步：征求意见期间就要先填登记表，已经开始但没完成的交易也算。所以该动起来的不是做 API 的，而是正在谈海外私有化部署、授权的，先把合同梳理一遍。

[Cain Wang | AI Infra & Agents @CainArch]

@leviqiao Good catch. I'd pin a canonical order per session on the client side (first-seen order, new tools appended), and log a hash of the tool block, so a cache-miss spike after a reconnect is easy to trace.

[Cain Wang | AI Infra & Agents @CainArch]

Gemini 4 Argon's intro price doubles on Dec 1 — same model, new sheet.

That's a reminder for agent cost models: sticker $/MTok is not $/task. Cache-hit rate and whether your system prompt stays byte-stable decide if the discount is real.

If your forecasts assume intro rates and 95% cache reads, stress-test them against the post-Nov-30 sheet before the cliff.

h/t @demian_ai

[Cain Wang | AI Infra & Agents @CainArch]

China just proposed adding 23 AI technologies (LLMs, agents, compute clusters, information processing) to its export-control list. Comments close Nov 15, but pre-export registration applies starting now.

What matters for anyone shipping models across borders:

Serving a restricted model via API, app, web or SaaS, or a lightweight on-device deployment: not a tech export.
Deploying a restricted model on-prem overseas: is an export. On your own overseas cloud, an automatic license applies.
Already open-sourced or published: out of scope.
The line is drawn at transferring the technology, not at selling access to it.

[Cain Wang | AI Infra & Agents @CainArch]

商务部、科技部 10 月 16 日发布公告，拟把 23 项人工智能技术列入出口管制目录，涉及大模型、智能体、算力集群和信息处理。征求意见到 11 月 15 日截止，但出口前登记从公告发布当天就开始执行。

我读完答记者问，整理了做 AI 出海最该搞清楚的几条：

用受限的大模型对外提供 API、App、网页、SaaS，或者做端侧轻量化部署，都不算技术出口。
把受限技术拿到海外做本地化部署，算出口；部署在自家的海外云上，可以走自动许可。
已经开源或者发过论文、进入公共领域的技术，不在管理范围内。
禁止出口的例子是 AI 驱动的恶意代码生成与变异；限制出口的例子是 AI 辅助漏洞挖掘。
需要登记的情形是：转让所有权、授权境外使用、技术的控制权转到境外主体。
我的理解是，管的是技术本身离开中国，不管把模型当服务卖出去。卖服务基本不受影响，卖权重、给授权、搞海外私有化部署的，要先查一下自己的技术在不在清单里。
咋感觉说的前段时间给Meta 的那家