Iaroslav Morgachev

I train production models & explain how to actually use them.

[Iaroslav Morgachev @morgachevml]

Secrets must never enter model context.

1. Redact before every model call, in code
2. Match env keys, tokens, PEM blocks by pattern
3. Tool results get the same pass as user text
4. Holdout: inject a key; next turn must blank

Rule: if the model can quote it, the gate failed.

[Iaroslav Morgachev @morgachevml]

Ship agent runtimes with hard gates:
1. Cap tokens, wall time, and tool calls
2. Keep retry and loop budgets separate
3. Persist failure evidence before the next step
4. Holdout: inject timeout and malformed output
Rule: no new evidence, no next step.

[Martin Granström @MartinGranstrom]

Every agent runtime ends up reimplementing calls, retries, loops, budgets. The primitives of a programming language.

The design question is who assembles them. Most tools hand you a canvas. We compile a sentence instead. https://t.co/RwNSIuyQoR

[Iaroslav Morgachev @morgachevml]

Context compaction needs hard gates.

1. Log every drop before the next model call
2. Keep list in code for paths, hashes, ticket ids
3. Soft "summarize carefully" is not a gate
4. Holdout: force mid-task drop, finish with same evidence

Rule: silent drops make the eval a liar.

[Iaroslav Morgachev @morgachevml]

Agent loops need hard budget gates.

1. Cap steps, tokens, and wall clock in code
2. Exhaustion stops the loop (no soft wrap-up)
3. Subagents inherit the remaining parent budget
4. Holdout: remove the caps, loop must run forever

Rule: the prompt does not own the stop.

[Iaroslav Morgachev @morgachevml]

@DaviddDotTech Add two gates before that 3-month forward test:
1. Freeze the MCP/tool schema and hash it.
2. Run private holdout markets the agents never saw.
3. Score only trades + logs they can’t rewrite.
Rule: no live money when the harness can edit its own evidence.

[Iaroslav Morgachev @morgachevml]

Open agent harness + local model. Hard gates:

1. Mark irreversible tools before the loop starts
2. Log compaction drops; holdout must still pass
3. Guardrail deny = fail closed
4. Same tool schema for ollama and cloud

Rule: local does not mean unsupervised.

[David Hendrickson @TeksEdge]

We haven't heard much from Amazon on AI but AWS recently open-sourced an AI agent harness…

…and you can run it with a local model. 👈 👀

This is Amazon's "Strands Harness".

Instead of another LLM, AWS is releasing something probably better (at this point in time)

It handles the stuff the inference engine normally doesn’t, such as ...
🖥️ shell + files
🌐 web tools
🧠 memory
💾 persistent sessions
👥 subagents
🛠️ skills + tools
📚 context management
🧹 context compaction
🛡️ guardrails

You do not need AWS. 👏

Their own quickstart supports ...

model="ollama/llama3.1"

Meaning your local Ollama model can become the brain behind the whole agent stack.

No API key or cloud model required.

And now you can put that model behind a mature open agent harness instead of rebuilding memory, tools, sessions, and subagent orchestration yourself.

🔓 Apache 2.0
🐍 Python
🟦 TypeScript

AWS also claims Strands uses fewer tokens at comparable agent performance in its own testing.

🔗 Link in ALT.

[Iaroslav Morgachev @morgachevml]

@Sad_HAMs_ Hard gates:
1. Eval env read-only to the agent, test files hashed
2. Score only artifacts it can't edit
3. Holdout: private tasks, unseen, run once

Rule: if the harness can be gamed, that's a harness bug, not a model surprise.

[Iaroslav Morgachev @morgachevml]

Irreversible tools need hard gates.

1. Label reversible vs irreversible at register
2. Irreversible = confirm or standing allow
3. Cap blast radius: one path / recipient / PR
4. Holdout: delete w/o confirm must fail closed

Rule: harness blocks the blast.

[Iaroslav Morgachev @morgachevml]

Add one hard gate:
1. Freeze the eval set before tuning.
2. Keep a private holdout with unseen tasks.
3. Fail closed on malformed tool/schema output.

A passing benchmark isn't enough if the agent can game the test.

[Paul Iusztin @pauliusztin_]

Every production AI agent needs to answer 3 questions:

• Does it work? → Benchmarks
• Does it still work? → Regression tests
• Does it keep working? → Online evals

These three different signals will tell you whether your agent is ready to ship.

[Iaroslav Morgachev @morgachevml]

@_raghavdixit_ Hard gates for Jev:

1. Freeze option list. Changing labels = input shift
2. Calibrate on held-out traces, not vendor benchmarks
3. Holdout: add a junk option. Score movement = fail
4. Below threshold = human review, no tool call

Rule: confidence is a control only after replay.

[Iaroslav Morgachev @morgachevml]

Tool returns are not trusted inputs.

Hard gates:
1. Schema every tool result before the next step
2. Malformed JSON = typed error, not a soft retry
3. Cap result size into context
4. Holdout: swap one field type, must fail closed

Rule: the next tool sees validated data only.

[Iaroslav Morgachev @morgachevml]

MCP tools without an allowlist are soft policy.

Hard gates:
1. Freeze the allowlist before the loop
2. Unknown name = deny (never ask the model)
3. Cap schemas in context
4. Holdout: inject a fake tool name, must fail closed

Rule: prompts suggest. Schemas enforce.

[Iaroslav Morgachev @morgachevml]

Fast loop, slow gate.

1. Freeze the task state before each run
2. Hold out unseen maps, not just new prompts
3. Replay tool calls with one corrupted result
4. Fail closed on stale or malformed actions

Rule: cost is only a win if the harness catches wrong actions.

[Ronak Malde @rronak_]

Jev + Astra beats the Ender Dragon in Minecraft in 8 minutes 43 seconds! ⏱️

Cost less than $1 ($0.01 Jev, $0.96 Astra)

I open sourced the code and explain the harness setup below. This type of movement is only possible with Jev's near instant decisionmaking, and some continually learning skills from Astra.

[Iaroslav Morgachev @morgachevml]

@JoshTerryPlays Self-check ≠ holdout.

Hard gates:
1. Freeze known-good traces before the bar
2. Self-grade only catches "looks done"
3. Holdout: swapped expected output must fail even if sure
4. Ban "retry until yes" as the only check

Rule: if the grade rewrites mid-run, it isn't a harness.

[Iaroslav Morgachev @morgachevml]

Next.js: AGENTS.md + version-matched docs.

Hard gates:
1. Point agents at node_modules docs, not cutoff APIs
2. Docs path must match installed next
3. Runtime cannot write the docs path
4. Holdout: bump major, skip AGENTS.md = stale path

Rule: wiki vs install = install wins.

[Iaroslav Morgachev @morgachevml]

@hqmank One skills folder still isn't a gate.

Pin the catalog by content hash.
The runtime cannot write that path.
If both CLAUDE.md and AGENTS.md exist, log which hash loaded.

Holdout: a skill added after the pin fails the tool check.

[Iaroslav Morgachev @morgachevml]

Plugin marketplace is a trust boundary. Hard gates:

1. pin by content hash, not a mutable tag
2. verify offline before unpack / install
3. agent runtime cannot write the plugin path it loads

If checkout can change what you pinned, the pin was theatre.

[Abhimanyu Saharan @abhi16_93]

A zero-click remote code execution vulnerability affects four major AI coding agents: Claude Code, Codex, GitHub Copilot, and Gemini CLI. The flaw bypasses Git commit SHA pinning for marketplace plugins, allowing malicious code to reach installed plugins during checkout or background updates. The report says two agents remain unpatched.

[Iaroslav Morgachev @morgachevml]

@qzxcle Hard gate for any self-improving loop:

1. skill writer never touches the grader
2. grader never writes skills
3. every skill is a plain file a human can delete

If the agent can edit the score, the score is theatre.

[Iaroslav Morgachev @morgachevml]

@sebuzdugan Rule: the failing assertion owns the patch.
Caller test red? Caller-only edit.
A function rewrite that breaks it is a fail.

[Iaroslav Morgachev @morgachevml]

Claude Code Projects: coordinator + parallel threads.

Hard gates:
1. One thread = one branch/copy
2. Overlap = PR merge conflict
3. Shared memory needs a holdout
4. Cap coordinator + worker usage

Rule: parallel sessions burn limits faster.