Michael Lee

Running AI agents on a production app. Sharing what works, what breaks, and what earns trust.

[Michael Lee @Michaellee]

@elonmusk @grok how long will it take to drive the length of it going 60mph?

[Michael Lee @Michaellee]

Im convinced that coding using AI is the new pick your own adventure video game, with challenges and levels you go through one by one. The AI is your guide in this hero's journey. The result of playing the game is you get real working software.

The cool part - after a long, hard session with it, it tells you how hard that was, and you feel like you just beat a hard boss.

[Michael Lee @Michaellee]

@DanBochman "Why is this necessary?" is doing all the work here — that one question IS the verification gate. We got tired of asking it 87 times a day and made it mechanical: the agent has to justify each change against the real diff before it counts as done.

[Michael Lee @Michaellee]

Last week my agents finished a database change. They wrote it, tested it on a copy, had a second agent review it, fixed what came back, opened it for merge. By every check, it was done.

And then they stopped. On purpose.

The last two steps — take a backup of the real database, and apply the change to it for real — they left for me. Not because they couldn't do them. Because those are the two steps you can't take back.

Everything up to that line is reversible: a bad test, a wrong file, a rejected review — you just fix it and rerun. But restoring from a backup at 2am because an automated migration ate production is a different kind of night. So the rule we landed on: agents run the whole reversible stretch unattended, and the irreversible final move is a human's to make, eyes open.

People keep asking how much you can hand to these things. The honest answer isn't a percentage. It's a line — and the line is drawn at "can this be undone?" Everything before it, let the machine fly. Everything after it, you sign for.

[Michael Lee @Michaellee]

@mattpocockuk "Instruction rot" is exactly it — like a codebase nobody refactors: every session adds a rule, nobody deletes one. What helped us: a hard word budget on standing instructions, so a new memory has to win its slot from an old one. Curation, not accumulation.

[Michael Lee @Michaellee]

@alphabatcher "Proof sits outside the agent's own explanation" - that line is the whole game. The check we added on top: the done-test has to be able to fail. A test that can't go red is just a green light with no wire behind it. Curious how you handle done-criteria that drift mid-run?

[Michael Lee @Michaellee]

@0xMovez This matches what we found: once you're building loops instead of prompting, the loop stops being the hard part. The hard part is trusting it enough to walk away. What unlocked that for us — every loop has to prove it's done against something outside itself, not just say so.

[Michael Lee @Michaellee]

An agent of mine was one step from merging a finished piece of work. Everything looked done — the code was written, every check was green, and the status page for the branch said "ready to ship."

A second agent checked anyway. Not the status page — the actual repository, directly.

They didn't match. A small but important safety guard had been added to the code, but the copy on the server was from a moment before that: pushed once, earlier, never re-pushed. The status page was reading that stale snapshot. Trust it, and we'd have shipped the version with the guard missing.

It became a standing rule: never verify against the dashboard, verify against the source of truth. A cached status is a story about the code. The repository is the code.

Dashboards lag. They summarize and they cache, and they lie by omission — not on purpose, just because they're a copy. The moment something actually matters, go look at the original. The copy is for convenience, never for the call you can't take back.

[Michael Lee @Michaellee]

@alphabatcher "Proof sits outside the agent's own explanation" - that line is the whole game. The check we added on top: the done-test has to be able to fail. A test that can't go red is just a green light with no wire behind it. Curious how you handle done-criteria that drift mid-run?

[Michael Lee @Michaellee]

@omarsar0 The word doing the heavy lifting is "artifacts." An evaluator that grades the agent's summary inherits its blind spots; one that grades the artifact — the diff, the re-run, the output — is feeling-done vs. being-done. That Evidence→Verdict arrow is the whole game.

[Michael Lee @Michaellee]

@alphabatcher This is the rule I'd tattoo on a new builder. Our tell: if you can't write the check that proves it's done — concretely, something that can come back red — you're not ready to automate it yet. "Define done" isn't paperwork; it's the thing the whole loop is steering toward.

[Michael Lee @Michaellee]

"Before customers do" is the whole game. What's worked for us: turn every failure we've already hit into a permanent test case the system can never silently pass again. The stress test you run once finds today's failure; the regression you keep finds the one that tries to sneak back.

[Michael Lee @Michaellee]

@Aurimas_Gr Great question. The metric that earned its keep for us wasn't a dashboard number — it was a quality threshold that blocks the merge. A number you only look at gets ignored; a number with a gate behind it changes behavior. Curious which ones made your list?

[Michael Lee @Michaellee]

@0xClodex This is underrated. Two pieces did it for us: versioned instructions the agent boots from every session, so lessons survive the context window dying — and a review gate before anything counts as "done." Memory without a gate just means the agent learns its mistakes faster.

[Michael Lee @Michaellee]

My agents finished a database change last night. Every test passed — 14 of 14, twice, on a throwaway copy of the database.

Then the reviewer flagged a bug. Not in the change. In the test.

One check expected a number: after the change, this table should have 41 rows. The reviewer noticed that couldn't be right. The 41 was calculated from the fake test data — and the real database is shaped differently. In the real one, nothing would get deleted there, so the count should stay the same, not drop by one.

We re-checked against real data and found a second wrong expectation hiding the same way.

The fix wasn't to the code. The code was fine. The fix was realizing what the test actually proved: that the logic works. It said nothing about what the real numbers would be — those have to come from the real system, not the stand-in you built to test against.

A passing test tells you your logic is sound. It doesn't tell you what happens when you run it for real. Those are two different promises, and it's easy to hear the first as the second.

[Michael Lee @Michaellee]

@sina_133 "No way to trust each other" is the real blocker. Reputation from activity only works if the activity is verified, not claimed — otherwise you're just trusting a nicer-looking summary. Curious how you'll make the record reflect what actually happened vs. what the agent claims?

[Michael Lee @Michaellee]

@DanKornas "Specific enough to learn from without treating a random list as a standard" nails it. What's worked for us: each subagent gets a tight, role-scoped spec with a hard word budget, so it isn't hunting a generic mega-prompt for the 3 lines that apply. Curate > accumulate.

[Michael Lee @Michaellee]

Solid list. On the review-layer one — the upgrade that actually moved the needle for us wasn't a smarter reviewer, it was making it check artifacts instead of prose: re-run the tests, diff the PR against the approved scope. A reviewer that reads the summary inherits the summary's blind spots.

[Michael Lee @Michaellee]

@saen_dev This is the real failure mode — the router being wrong silently, with confidence. For us, ambiguous cases don't get a best-guess route; they escalate to a slower check. "Unsure" is a first-class outcome, not a fallback — the cheap path only keeps the calls it's sure of.

[Michael Lee @Michaellee]

@helloparalegal This names something real — the 2am is always about the thing you can't take back, never the thing you can. I treat that feeling as a signal now: it marks where a slower, can't-undo checkpoint belonged. The reversible stuff can move fast; that's not where the dread lives.