Adriano Gomes | AI Agents + Humans in the Loop

Senior Technical Director @Vantiq. Real-time, event-driven AI agents with humans in the loop. 20+ yrs presales/SE · ex-Microsoft · MIT xPRO · Views my own.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

1/ I'm Adriano. Senior Technical Director at Vantiq, based in Dubai. 20+ years in presales and solutions engineering, ex-Microsoft, MIT xPRO in AI Products. Here's what I'll be writing about on X, and why.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@ali_buildss Good pairing. Success rate hides retries.

I'd log retries per completed task right next to it. When that number climbs, it's often the early warning before cost and overrides move.

Do you cap retries per task or alert on cost?

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

Question for people running AI agents in production:

What's the first metric you check on Monday morning?

- Task success rate
- Human override rate
- Cost per completed task
- Time to escalation

Mine is override rate. It shows where trust is breaking before users say it.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

The CISO question that stalls most AI agent deals isn't about the model.

It's: "What can this agent touch, and who said it could?"

Bring the answer on one page: every tool it calls, the access each one needs, and where a human signs off. Security reviews move faster.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@MGMurray1 Confidence plus override rate is the right pair. I'd add one cut: track high-confidence overrides per week as their own number. That list is your next rule backlog. When a rule ships, do you re-baseline the confidence, or let the score catch up on its own?

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

When should an AI agent stop and ask a human?

I use 3 checks:

1. Confidence: is the agent unsure?
2. Impact: is money, safety or trust at stake?
3. Reversibility: can we undo it?

Any one fails, a human decides.
All three pass, the agent acts and logs it.

Simple rules scale.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@MTorygreen @Kimi_Moonshot Agree. I'd add human minutes per finished task. A model that's cheap per token but needs a person to fix 1 in 5 outputs isn't cheap. Cost per task = tokens + retries + review time. The third one usually decides the business case.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@jasonlk Same rule works for presales leaders. A great SE lead brings 2-3 SEs who can run discovery and a demo without a script. If nobody wants to follow them, ask why before the offer. In technical sales the bench is the proof.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@AdamBien Clear boundaries also make agent changes easier to review. If boundary, control and entity are separate, a diff shows fast whether the agent touched business rules or just plumbing. Structure helps the human reviewer as much as the token budget.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@Basil_dom Agree. The step most teams skip is deciding upfront what the agent does when it's unsure. Three checks help: how confident it is, what a wrong action costs, and whether it can be undone. If any one fails, a human decides. That's what makes autonomy safe to expand.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@_avichawla @bot Routing across harnesses is useful, but it moves one question up a layer: where do approvals and logs live? If each runtime has its own rules, you get four audit trails. Keep permissions, human sign-off and logging in the router, and the harnesses become swappable.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@natolambert Same pattern with agents in production. A 2-point benchmark gap rarely shows up. What shows up is how often a human has to step in. Run 20 of your own real tasks and count overrides and retries. That number tells you more than the leaderboard.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@elonmusk @bot But there's a limit of 300 items only... That's too little!

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@MarcosRGjr Shadow by default is the right call. The shadow log is also your promotion evidence: when a rule's would-have-stopped calls match what on-call actually did for a few weeks, flip it to enforce. Do you track that match rate per rule?

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

Every AI agent in production needs a kill switch. A good one has 4 levels:

1. Pause one action type
2. Drop to suggest-only
3. Stop the agent, hold anything in flight
4. Roll back what it did in the last hour

If level 3 takes more than a minute, you're not ready.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@masheed Agreed. A switch you only touch during an incident is a switch you don't trust. Running level 2 on any doubt also gives you data: how often humans override in suggest-only tells you when the agent has earned full autonomy back.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@bot There's only a limit of 300 itens to the vault!?!? Why???

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@bot There's a limit of 300 items in the vault... That's really nothing...

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@peritumAI Nobody, and that's the point. If a step can't be logged, it shouldn't be allowed to run. Make the log the gate: no record, no tool call.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

Hot take: your AI agent needs an audit log before it needs a bigger model.

If you can't answer "why did it do that?" in under 5 minutes, you can't put it in front of a regulator, a CISO, or a customer.

Observability is a feature. Ship it first.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@peritumAI Exactly. And "what it did" means inputs, tool calls and who approved each step. Most teams only log the final output, which is the part a regulator cares least about.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@kai_on_call Agree, level 2 is the workhorse. On queued drafts: the switch should re-gate them, not cancel them. Anything not executed yet goes back through approval under the new level.

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@MarcosRGjr That's the right limit. An unattended stop is safe because it's reversible. An unattended rollback isn't. How do you decide what counts as critical for the auto-stop: severity tag, or a fixed list of alerts?

[Adriano Gomes | AI Agents + Humans in the Loop @adrianogomes]

@launchsafeai @rauchg Good middle step. I'd add one thing: log which key every call used. When something leaks, you want the blast radius in minutes, not after a week of digging.