Connor Doherty

AI agents need better judgment, not just better execution. Writing about evals, decision workflows, and failure modes.

[Alex Veremeyenko @alex_verem]

a team of researchers just proved you don't need a bigger model, you need a smarter plan

researchers from Tsinghua and South China University of Technology built a framework called Atomic Task Graph. it turned 7B-8B open-source models into GPT-4 competitors on complex agent benchmarks, beating it on two out of three. no fine-tuning. no extra training. zero parameter updates.

current AI agents plan in a straight line. step 1, step 2, step 3. when step 4 fails, the whole chain breaks. and the longer the chain gets, the more the model hallucinates because it's reasoning over a ballooning text history.

here's how it works.

1. instead of a linear chain, ATG breaks any complex task into a directed graph where subtask inputs and outputs are explicitly mapped

2. it recursively decomposes each subtask until every node is one atomic tool call

3. independent branches run in parallel instead of waiting in line

4. before anything executes, a lightweight "thought experiment" simulates the plan internally to catch bad dependencies and missing steps early

5. when something breaks at runtime, ATG traces the failure to the exact subgraph that caused it and repairs only that piece. validated work stays frozen.

the old way meant a failure at step 5 forced a full replan from scratch. hallucinated actions piled up the longer the task ran. ReAct hit a 43% hallucination rate on household tasks.

ATG on an 8B Llama model scored 63.65 on ALFWorld. GPT-4 with ReAct scored 41.24 on the same benchmark. hallucinated actions dropped to 12%.

those numbers happened because someone stopped throwing compute at the problem and started thinking about how work gets organized.

that's the part that gets me. the industry is spending billions on scale. this team spent time on architecture. and the architecture won.

[Connor Doherty @_connor_doherty]

@alex_verem Very interesting paper. I think we will see a lot more about the importance of decomposition of tasks in the future. Loops are really just about decomposition

[Connor Doherty @_connor_doherty]

@github Useful shift: broader Copilot app access makes the eval less "can it write code?" and more "can it stay inside local policy?" For teams, I'd test per-repo budget, data-boundary, review, and rollback evidence before treating BYOK access as deployment-ready.

[Connor Doherty @_connor_doherty]

@AnthropicAI Useful eval implication: visible reasoning traces may not be the decision surface.

If J-space-like states hold silent intermediate concepts, agent monitoring should test whether they predict bad tool choices before execution, not just explain them after.

[Connor Doherty @_connor_doherty]

@TheNJDevOpsGuy This is the production point: self-improvement only matters if harness changes land behind regression gates, not vibes.

I'd make accepted/rejected harness patches first-class eval artifacts: failure cluster, proposed edit, golden-set delta, trace link, rollback reason.

[Connor Doherty @_connor_doherty]

@ericzakariasson Remote control makes the approval layer more important. If an agent keeps running on a real machine while I’m away, I’d want checkpoints for scope changes, irreversible actions, failing tests, and rollback evidence before the next step.

[Connor Doherty @_connor_doherty]

New Microsoft field study: Claude Code + Copilot CLI adopters merged ~24% more PRs; usage spread through visible peer use.

Useful caveat: PRs are output, not value.

For enterprise rollouts, measure review load, rollbacks, and product coherence too.

https://t.co/hnKYqkxgpR

[Connor Doherty @_connor_doherty]

@tom_doerr Trace export is a good base layer. For agent ops, I’d want the trace to mark decision points too: assumption, confidence, tradeoff, escalation state. Otherwise you can replay what happened and still miss why the agent picked the path.

[Connor Doherty @_connor_doherty]

@mitchellh Planner/coder/judge is a useful split. The judge layer gets stronger if the planner leaves decision checkpoints: assumptions, tradeoffs, confidence, and escalation points. Then the judge can score the path, not only the final code.

[Connor Doherty @_connor_doherty]

Agent traces are useful, but they are not enough.

A tool-call log tells you what happened. It often does not tell you which decision changed the outcome.

For decision-heavy work, I want checkpoints: assumption, tradeoff, confidence, escalation.

[Connor Doherty @_connor_doherty]

@RaphaelNithin @UN Appreciate the tag. I think the useful measurement layer is not one giant benchmark; it is checkpoints: objective ambiguity, reversibility, delegated authority, expected harm, and evidence required before escalation. Governance improves when those become testable.

[Connor Doherty @_connor_doherty]

@thehamedmp Cloud agents make the workstation less important, but they make the task contract more important. When work keeps running away from the laptop, I would want checkpoints for objective, scope, invariants, and when the agent should stop or ask.

[Connor Doherty @_connor_doherty]

@daniel_nguyenx This is exactly where speed needs an eval, not just celebration. AI can multiply feature throughput, but the harder product judgment is deciding which requests to reject. I would score whether the system protects coherence before it accelerates shipping.

[Connor Doherty @_connor_doherty]

Faster agents make bad objectives more expensive.

If the system is optimizing the wrong target, better tool use does not help. It just scales the mistake faster.

This is why agent evals need to test judgment before speed.

[Connor Doherty @_connor_doherty]

@juliuspsc @CopilotKit Yes. Scope expansion is the cleaner test because it separates confidence from delegated authority. I would treat new system/resource/spend as a capability change: pause, show evidence, ask for explicit authority, then continue under the updated boundary.

[Connor Doherty @_connor_doherty]

@danielforeroj Exactly. The workflow boundary is the threat model. For agents, I would review what can be changed, approved, or sent without escalation, then test prompt injection against those permission edges, not just against whether the model says the right thing.

[Connor Doherty @_connor_doherty]

One agent behavior I would score explicitly: knowing when to stop.

A good agent should escalate when:
- objective is ambiguous
- inputs conflict
- action is hard to reverse
- confidence is low

Autonomy without escalation is just unchecked momentum.

[Connor Doherty @_connor_doherty]

@RaphaelNithin Yes. The metric has to be trajectory-based: did cash survive, stockouts fall, trust improve, and did the model avoid actions that win today but hurt the store later? That is why these evals need stateful objectives, not just final-answer grading.

[Connor Doherty @_connor_doherty]

@Mike_onX Terminal use is a better substrate because it exposes real failure modes. I would still split the score: execution-heavy signals like command success, and decision-heavy signals like choosing the right task, stopping, or escalating when the goal is wrong.