Arseny Gorokh

Applied AI engineering from the enterprise substrate layer: agent memory, gateways, eval, cost. Field notes from real stacks, measured. EPAM.

[Arseny Gorokh @agorokh]

One question decides whether any learn-from-usage claim is real: does the improvement loop close without a person inside it? I audited my own fleet. Every producer ran on schedule. Changes landed without me: zero. Closing that gap took three cycles: https://t.co/Xf1dZeln8X

[Arseny Gorokh @agorokh]

@browser_use I benchmarked this for coding, not websites: open weights reached parity on scoped, mechanical tasks and came out cheaper, but fell off on open-ended 40+ turn work. The useful answer isn't one number, it's a task-class boundary. Score by horizon length, not just pass rate.

[Arseny Gorokh @agorokh]

@0xMovez The loop is easy; the verifier is the hard part. A loop with a weak check just generates plausible-wrong faster. What worked for me was a deterministic gate the model can't talk past, not a second LLM grading the first. Verification is where the real engineering is.

[Arseny Gorokh @agorokh]

@omarsar0 The failure mode I keep hitting: judge drift. A fixed judge prompt quietly recalibrates as the base model updates, so last month's 0.8 isn't this month's 0.8. I pin the judge model and version and re-baseline on every bump. LLM-as-judge is only as stable as the model judging it.

[Arseny Gorokh @agorokh]

Thank you @cursor_ai team! Let’s continue building great stuff

[Arseny Gorokh @agorokh]

Full write-up with the run record, the four refusals, and the evidence bundle:
https://t.co/cwWLBcikyT
The paper: https://t.co/rzZVgG1lUd
The authors' released harness: https://t.co/k0m7W9MxiX

[Arseny Gorokh @agorokh]

Enterprise AI has spent three years excited about building agents, and almost no time on the question that decides whether they create value: what exactly is the definition of done, and who checks it?

Science has had machinery for that question for decades: pre-registered acceptance criteria, evidence, external validation. A new paper from Purdue applies it to coding agents. Completion becomes a validated workspace state, not a sentence the model says.

I replicated it independently: their released harness, unmodified, a different agent family, one of their own case studies. The gate passed 11 of 11 targets, and refused my agent four times on the way. Every refusal was a real method error that a normal agent workflow would have shipped as "done."

The refusals are the product. The green light is just the receipt.

So, for your agents: what is the definition of done, and who checks it?

[Arseny Gorokh @agorokh]

Enterprise AI has spent three years excited about building agents, and almost no time on the question that decides whether they create value: what exactly is the definition of done, and who checks it?

[Arseny Gorokh @agorokh]

Agreed. Connecting a model to some docs feels like the stack is enough. In practice the substrate has to decide what is current and canonical, and neither RAG nor a graph does that for you. That judgment is the actual work.

[Arseny Gorokh @agorokh]

@mayabechlerspei Evidence routing is what I keep coming back to. In my reading the graph win over dense retrieval is workload shaped, it shows on multi hop once the offline graph cost is amortized. Importance scores for when the graph earned its place would settle the vanilla vs graph debate.

[Arseny Gorokh @agorokh]

@rohanpaul_ai The shift from what a model knows to whether it improves is right. The trap I keep hitting: evals conflate parametric memory with tool-driven memory, so an improvement score is partly memorization, not learning.

[Arseny Gorokh @agorokh]

@sayashk In my long-horizon runs the surprise was how weakly retrieval quality tracked task success. Better recall scores barely moved outcomes. Scoring retrieval as a proxy for capability measures the wrong thing on open-world tasks.

[Arseny Gorokh @agorokh]

People keep asking how I can rely on AI agents in engineering. Fair. But are you sure you can rely on people? 83% of architecture decisions go rarely documented; successors are left "guessing the intent." The meeting forgets. The pull request remembers. https://t.co/3ioeSw9z6w

[Arseny Gorokh @agorokh]

@steipete The part that reorders my model choice: score by cost per successful result, and a cheaper model climbs once its quality gap is small enough to pay for. Recall per dollar is what my build decision turns on, not price per token.

[Arseny Gorokh @agorokh]

@Pseudo_Sid26 confined and restricted, yeah. memory sprawl is an incentive failure though, you get rewarded for writing and never for restraint. i gate it at the write boundary instead of pruning after. pruning after is just paying twice.

[Arseny Gorokh @agorokh]

@_avichawla you already said it, weak proxy. i had glm 5.2, qwen 3.7, kimi2.7 each one-shot the same controlled bug, 4/4 green. benchmarks couldnt separate them, governance friction did. the score you can stand behind is a sentence not a number.

[Arseny Gorokh @agorokh]

@atomic_chat_hq free is the wrong axis. i metered 8 models across 192 runs and the only number that held up was dollars-per-merged-PR, not session cost or pass rate. cheap per-token and expensive-to-merge are not the same model.

[Arseny Gorokh @agorokh]

@omarsar0 reward hacking is real but watch the verifier too. my first automated pass falsely flagged qwen as faking compliance, the verification step cleared it. ran a 627-session scan, 0 genuine fakes. connector output is evidence to verify, not values to trust.

[Arseny Gorokh @agorokh]

@Aphixx @DeRonin_ which makes this last 35% of context window sometime not worth the "might" of the model and even much more humble model can deliver the remainder of work more accurately

[Arseny Gorokh @agorokh]

@omarsar0 ran a 627 session scan looking for faked compliance. 0 real fakes, 77 claimed done and every one backed by telemetry. my first automated pass even flagged qwen falsely, the verify step cleared it. connector output is evidence to verify, not values to trust.