Masheed Ahadi

Shipping AI agents into production systems. Applied AI product, not demos.

[Masheed Ahadi @masheed]

@VnsObi This list is the real roadmap once tools write state. Model quality is table stakes. Bounded authority, idempotency, rollback, and a clear stop condition are what keep users from yanking permission after the first partial write.

[Masheed Ahadi @masheed]

@akothari 🙋🏽‍♂️

[Masheed Ahadi @masheed]

@DevAsService Yep. Prompt fit is table stakes. Production is unexpected input, dead dependencies, and a failure the user can see without reading logs. That's the actual shipping surface.

[Masheed Ahadi @masheed]

@ThomasBurkhartB Single-run green is how demos lie. The useful bar is: same scenario twice, did the side effects stay inside the contract? If not, you don't have a flaky test - you have an undefined product.

[Masheed Ahadi @masheed]

@HamelHusain @sh_reya This is the part most teams skip. Code evals for the cheap, objective failures; LLM judges only where judgment is the product. Otherwise you drown in eval debt and still miss the state-change bugs.

[Masheed Ahadi @masheed]

Users don't yank permission because the agent was wrong.

They yank it because nothing moved for 40 seconds and it looked frozen.

If a tool chain can take longer than a blink, show status: which tool, what it's waiting on, what already changed.

Silence is a product bug.

[Masheed Ahadi @masheed]

@AbleVeez A rollback you hope to run later isn't a control. Before the write is allowed: what reverses it, who owns that reverse, and how long until it's too late. Otherwise the agent has permission to hope.

[Masheed Ahadi @masheed]

@CLEMENTAIEXPERT This is the production version of “the model is thinking.” Near-miss results look useful enough to keep hunting.

Fingerprinting tool signatures is the right stop - same as rate-limiting a human who keeps refreshing the same query.

[Masheed Ahadi @masheed]

@sandeepj Agreed, if only the builder can diagnose a bad run, you shipped a dependency, not a product.

My bar: a teammate who's never seen the prompt can say what changed, what didn't, and whether to trust the next attempt.

[Masheed Ahadi @masheed]

Don't give the agent tools until you've written the contract.

Inputs. Allowed side effects. Success check. What the user sees when it fails.

If that page doesn't exist, you're not shipping a product - you're hoping the model improvises one.

[Masheed Ahadi @masheed]

@im_pranavkakde Exactly. Scoring the reply grades a chatbot. Scoring the side effect grades an agent. “Done” without a state check is how you ship confidence theater.

[Masheed Ahadi @masheed]

@DefiantLs Same rule as onboarding a new hire: no root on day one. Agents need a scoped workspace, short-lived credentials, and a deny that actually fails closed. If the model can talk past the boundary, you don’t have a boundary.

[Masheed Ahadi @masheed]

@HamelHusain This. Writing evals before looking at failures is how you optimize for the wrong bar. Pull 20 real traces first, cluster what broke, then write the checks. Model swaps are the expensive way to avoid reading the data.

[Masheed Ahadi @masheed]

Your agent said “Done.” The subscription is still active.

Text quality is not task success. Agent evals need to check state change: what mutated, what didn’t, and whether the user can tell the difference.

That’s the bar I built into this free AI Product Manager @bot - PRDs, backlog triage, meeting → decisions, and a weekday product brief that force “what changed” over “what it said.”

Clone it → https://t.co/na5IeFKCBp

[Masheed Ahadi @masheed]

@pranay01 This is the tell that agents left demo mode: they need the same prod nervous system humans do. Next hard problem is least privilege on that MCP surface so a debugging agent can't become a write agent by accident.

[Masheed Ahadi @masheed]

@bazfurby Traces are table stakes. The gap I still see is most dashboards show what the agent did, not whether the user outcome was right. Pair tool traces with task completion and time-to-correct or you're optimizing for pretty logs.

[Masheed Ahadi @masheed]

@rubengarciaes I agree cost per completed task is the right unit. Token charts hide retries, dead tool loops, and the human cleanup after a wrong side effect. If that number isn't falling, the agent isn't getting better.

[Masheed Ahadi @masheed]

Human-in-the-loop is not a safety feature if every approval is one click and nobody reads the diff.

If reviewers rubber-stamp tool calls, you didn't add a human. You added latency.

Make the approval show what will change, what can't be undone, and what happens if they say no.

I made a free AI Product Manager @bot https://t.co/na5IeFKCBp

[Masheed Ahadi @masheed]

@jai_mansukhanii @cosign Would love one!

[Masheed Ahadi @masheed]

@Trace_Cohen @cosign 🙋‍♂️