Joe Zhou

AI engineer writing about the real economics of AI agents—models, infrastructure and production failures.

[Joe Zhou @joezyai]

The model bill is not the AI system bill.

Tokens are simply the easiest cost to see.

The real bill includes everything the model causes: tool calls, retries, infrastructure, human review, maintenance, and failures.

Optimizing token price optimizes one line item—not the system. 🧵

[Joe Zhou @joezyai]

@ba_niu80557 这是您拍的吗？很棒呐

[Joe Zhou @joezyai]

@Idolevy31 78% → 90% evidence recovery at the same retrieval budget is the comparison I want more of.

Equal-spend evals beat another "we upgraded the model" post.

[Joe Zhou @joezyai]

@im_pranavkakde @ddyods Yeah — schema checks first, cheap judge next, big model only when it's actually ambiguous.

We've wasted a lot of spend putting the expensive judge on every row.

[Joe Zhou @joezyai]

@horizon_bench Same model, same test, 23 runs, scores from 58% to 77%.

At that point the screenshot is just a mood board.

[Joe Zhou @joezyai]

@anilpervaiz Held-out at 90.5% after 78.6% on the first cut is the sequence I trust.

Separating tool/API failures from model variance also saves a lot of bad revert calls.

[Joe Zhou @joezyai]

@Vtrivedy10 This is the part most bench decks skip.

If the eval doesn't look like the work your agents actually do, you're grading a different product. The scoreboard can look great. Transfer is the whole point.

[Joe Zhou @joezyai]

@pauliusztin_ $40 and she still had the moral high ground.

Mine once re-ran the same eval three times because a timeout looked like a flaky judge. The invoice was fine. My face was not.

[Joe Zhou @joezyai]

@HamelHusain Yeah. We kept skipping that step.

The workflow wanted labeled cases first. The traces we already had were screaming which failures mattered. We spent a week "correcting" a grader before we'd even looked at the data.

[Joe Zhou @joezyai]

@fankaishuoai 请问您为什么不考虑TikTok呢？短视频不是这个最大吗？

[Joe Zhou @joezyai]

June paper: 125 adversarial actions. Reviewers only agreed "risky" at κ=0.52.

Tired reviewers flipped the curve. More escalation, worse safety.

Human review is a queue. You don't staff your way out of a bad agent with another checkbox.

https://t.co/MzzLgzfLFz

[Joe Zhou @joezyai]

@lizengco Kids know negotiation very well. They also know how to test your red lines 😂.

[Joe Zhou @joezyai]

Same model. Two providers. Up to 2.6× on the invoice once cached input is actually billed.

That's from a cache-aware routing writeup this month. List-price routing picks the wrong one.

I want the gateway to see cache, not just the sticker.

https://t.co/FbNALyOFZs

[Joe Zhou @joezyai]

Claude Fable 5.1 cut cache-read price 75%.

Artificial Analysis still has it at $3.76 per Intelligence Index task. Fable 5 was $3.14. About 1.7× the output tokens.

Cheaper reads. Higher bill per task.

https://t.co/C3Th2BENkh

[Joe Zhou @joezyai]

@fankaishuoai 如何理解Instagram 有潜力？

[Joe Zhou @joezyai]

99.3% cache hit. API cost down 88.6%. Effective $0.57 per million tokens.

July inference-economics paper. Under their shared allocation, that beat a $2.83 on-prem unit.

Caching doesn't sit on the model price sheet. It sits on the bill.

https://t.co/g3tojFEg4q

[Joe Zhou @joezyai]

Anthropic's multi-agent researcher beat a single Opus 4 by 90.2% on their eval.

It also chewed about 15× the tokens of a chat. Token spend explained roughly 80% of BrowseComp variance.

I want that accuracy. I also want the 15× metered before we call it cheaper.

https://t.co/WXP7JQptRa

[Joe Zhou @joezyai]

@runbywren The separation was not very clear in the paper.

[Joe Zhou @joezyai]

A coding agent can burn about 3500 times the tokens of a one-shot on the same job.

April paper. Same task, different runs, up to 30× spread. Accuracy didn't ride with the token count.

The invoice can look like quality. It isn't.

https://t.co/4tcvbRruVc

[Joe Zhou @joezyai]

@use_kibble I'd hope the team will deploy one version of the harness into production, so that is the canonical.

For experimentation, you can leverage the registry to record configs. Of course, in real day-to-day, things are messy.

[Joe Zhou @joezyai]

https://t.co/KdM7vhfHsq

[Joe Zhou @joezyai]

Same model. They only changed the harness. Cost per task went $0.21 to $0.12.

July paper (Harness Effect). Tokens per task: 14.2k down to 8.8k. Nobody swapped the lab model.

I keep getting asked which model to buy. I'd rather know which harness ran.
https://t.co/fz8u42dUxe