@czioutas Strong case for that split. Use agents where ambiguity is real; make repeatable decisions deterministic. Measure cost per accepted outcome, including retries, review and recovery. A cheap run that leaves human repair work has not saved money.
@AamirAnsar94694 Put an authority boundary between planning and tools: who may approve which action, under which dated rule, and can revocation stop a run? Observability tells us what happened afterward. The harder production test is stopping an out-of-scope action before it happens.
@paleknight The $0.57 median is interesting, but is that per attempt or per accepted outcome? User reactions get us closer than lab tests. I’d still count failures, retries and human correction. Otherwise cheap sessions may conceal expensive work.
@ruansherry A useful test: a real workflow where a rule changes midway. Record the source date, authority, decision and accepted outcome. Does the agent notice stale context, stop and replan? Otherwise evaluation may reward fluent completion over dependable work.
The coming AI IPOs will tell us more about financing intelligence. Buyers also need to know what survives a provider failure: current rules, open cases, evidence and a way to hand the work over. An agent continuity plan needs more than software escrow. https://t.co/ACiGV1uoWQ
@cv_usk Simulated users are useful for finding regressions, but I would keep a fixed set of real task cases alongside them. Otherwise an agent can get better at satisfying its simulator while missing a changed rule or a difficult handoff. What is the pass bar for an accepted outcome?
@ArtificialAnlys@cognition Useful to see both models' API spend counted. The next comparison I'd want is cost per accepted change: same tests, same human review, retries and repair time included. A cheaper benchmark run can still be dearer if the handoff leaves work for the maintainer.
@Atlassian If AI use becomes the KPI, teams will optimise for visible prompting. I'd measure the accepted change instead: defect rate, time to diagnose failures, and whether another engineer can maintain it. The last measure catches technical debt that a faster first draft can hide.
@cizotech The useful test is cost per accepted outcome, not model price alone. Retries, tools, review and idle capacity can erase a cheap-token saving. A sovereign model option also needs proof of where data and work went. Build now, but measure the whole job.
@RV_Smirnov@Thom_Wolf@yoheinakajima Agreed. The test is whether identity, permissions and evidence travel with a job across providers and tools. Can the buyer show who authorised it, where it ran, what changed and why the result was accepted? Separate standards only help if that record joins up.
@martinrunge@Atlassian Identity and visible presence help. The harder test is authority across tools: when an agent moves between Jira and Confluence, which system checks the task, current permission and result? A shared trace should let the team challenge the action, not just watch it happen.
@kannagoldsun Every source can be accurate yet wrong for an action if its effective date, jurisdiction or purpose is lost. I’d check those at the tool boundary, with target, authority and recovery. Lineage should prevent a bad write, not just explain it later.
A UK-hosted application can still send an agent’s work, logs or support access across borders. If a contract limits where data may be processed, what evidence proves the route each job actually took? https://t.co/jZleeJDdp0
The routing tree is useful, but “fewer tokens” is only an input measure. I would record the cost of each accepted change: all model calls, retries, tool use, review time and defects found later. A cheap scout plus an expensive reviewer can win; so can one stronger pass. The comparison needs the same task and acceptance test.
The hard part is delegated authority. If one agent brings its memory into a shared room, which instructions and data may the others see, and who can revoke access mid-task? I would test a partner leaving while work is queued: can the system stop new tool calls, account for unfinished actions and show a trace of what crossed the boundary?
@arosplatforms@jetbrains A useful extra gate is the exit drill: can another engineer switch the backend, reproduce a failed task and restore service? I’d include idle capacity, upgrade work and recovery time in the cost comparison. Local ownership still needs an operating budget.
@al3rez I’d keep the acceptance tests and promotion rules outside the agent’s editable harness. Then test whether a “better” version still respects permissions and can roll back safely. Who owns the decision to promote a candidate when quality improves but recovery gets worse?
@StatsWire Is that cost per attempted task or per accepted result? I’d want failed attempts, retries and review time in the same comparison. A lower price is useful; the buying decision needs the cost of getting the same work accepted at the same quality threshold.
The hidden danger isn’t necessarily a rogue AI. It may be dependency growing faster than our ability to understand, afford and govern it.
New article on agent standards, maintainable code and the practical ability to stop, repair and replace. https://t.co/OpsKq7qDUF
@Pradeep891730 Does the evaluation restrict the agent to what was knowable at each decision date? A later report can make an earlier decision look obvious. A time-frozen evidence set plus checks written independently of the dataset would make this a stronger test.