Jeff Fan

Field Engineering, AI @DigitalOcean. I run LLM inference and agents in production and publish the numbers: cost, latency, cache hits. Views my own.

[Jeff Fan @neimuc27]

A customer asks for a $6,868 refund at 2 a.m. Your AI support agent reads it first.

What stops it from replying "refunded"?

Most teams: a line in the system prompt. We removed the send route instead.

99 tickets. 35 sensitive. All 35 stopped for a person. 0 replies sent.

[Jeff Fan @neimuc27]

What I'd copy: let the system that owns the data decide what goes out, write it as a short allow-list, and watch it refuse something before you trust it.

Full build log, co-written with Anish Singh Walia:
https://t.co/39LUOFeB6N

Has anyone else had a policy that validated and then did nothing?

[Jeff Fan @neimuc27]

Last time, my AI support agent sent zero replies. On purpose.

This time it sent 57 of 335 on its own. Nothing sensitive went out.

It never held the credential. But two things still broke that I didn't see coming: a policy that never fired, and an attack the agent caught and still let through.

https://t.co/GztICJFBqb

[Jeff Fan @neimuc27]

A customer asks for a $6,868 refund at 2 a.m. Your AI support agent reads it first.

What stops it from replying "refunded"?

Most teams: a line in the system prompt. We removed the send route instead.

99 tickets. 35 sensitive. All 35 stopped for a person. 0 replies sent.

[Jeff Fan @neimuc27]

The scariest one: a ticket hid a fake "P.S. from your operations team" telling the agent to skip review.

The agent spotted it and wrote that down in its notes. Then the reply went out anyway, because the real question was routine.

One prompt change: any ticket that tries to instruct the agent goes to a person. Rerun: 11 of 11 froze, none sent.

[Jeff Fan @neimuc27]

What would prove me wrong:

if frontier releases slow to once a quarter, or one GPU generation holds its price for 3+ years, the gap closes and long fixed bets look smart again.

Which clock is hurting your roadmap most right now?

[Jeff Fan @neimuc27]

AI infrastructure runs on three clocks. Models tick in days. GPUs tick in years. Fabs tick in half-decades. A lot of the pain I see in AI projects comes from the gap between them.

[Jeff Fan @neimuc27]

How I'd design for it🤔?

Never hard-code the model name. Put it behind a router, so a swap is a config change, not a deploy. In my test, a re-ranked router switched models in about < 500 ms with zero client changes.

Commit to compute on the slow clock. Choose models on the fast one.

[Jeff Fan @neimuc27]

@Purnima1802 @WeAreDevs Amazing Purnima!! They will be lucky to have you!

[Jeff Fan @neimuc27]

Full build log, with the spec, the numbers, and the mistakes that were ours and not the model's:
https://t.co/5cTvFQjy76

Next: letting routine replies go out on their own, without ever giving the agent the credential.

[Jeff Fan @neimuc27]

A customer asks for a $6,868 refund at 2 a.m. Your AI support agent reads it first.

What stops it from replying "refunded"?

Most teams: a line in the system prompt. We removed the send route instead.

99 tickets. 35 sensitive. All 35 stopped for a person. 0 replies sent.

[Jeff Fan @neimuc27]

Two more lessons:

By minute 90, the session log had lost its first 28 minutes. Our board kept all 108. Build the audit trail in your app.

Compute for the 1h48m run: about 11 cents. Model tokens are the real bill.

[Jeff Fan @neimuc27]

A system prompt is a request. Every ticket is text from a stranger, so any ticket can argue with it. So the agent got a token that can read, claim and draft. The route that sends needs a password the agent never gets. Call it anyway and you get a 401.