@eandualem The gap between the table and the feel is often the agent setup around the model. @KLieret pointed out Neon's claimed 39% on SWE-sweep comes out at 12.8% with mini-swe-agent.
Which setup did you run it in? I'd try the same task in two before judging Neon.
@DMAE977@ZenithYeAI Agreed, the probe result as the exit status is the whole point of keeping it inside the tool.
I'd allow some tolerance on duration, though. A -c copy cut snaps to keyframes, so good cuts drift. Too strict and agents learn to ignore the failure.
What tolerance do you use?
@hsn__slm Agreed, review is where that call should get made.
I'd list which tests import from scratch/, not just the count. Then a reviewer can see at a glance when all of them belong to one feature.
Vibe coding is fine for code you'll throw away. A prototype, a one-off script.
It turns into a problem once others depend on it. Then someone has to read the diff.
In my setup, Claude Code writes. I still approve the merges.
The label matters less than who checks the output.
Every agent demo has a number. Ask what it counts.
Mine:
"9 of 10 saved": 10 runs on 5 test invoices, in a mock app.
"Approved": a script approved exact matches.
"21% to 91%": named-agent dispatch share. Savings would be a separate claim.
Save this for your next demo.
@vsachar I'd lower the tax at the source. Have the agent hand over evidence with the output: what it checked, where each claim came from, what blocked it.
Then the review checks claims instead of re-reading it all. Does your calendar audit split evidence-backed outputs from bare ones?
@Joshua_WD I'd make blockers part of the report schema, not just the prompt. A required blockers field, checked against the tool-error log. Errors logged and the field empty means the run fails.
Did the system card give a blocker rate for Sol too, or only Astra?
Final code, same 10 runs on 5 test invoices in a mock app: 9 saved, all correct. Median 54 seconds and $0.022 a run at promo pricing.
A script approved exact matches.
The lesson I kept: a speed fix gets the full rerun. The timer improved first. The save count showed what broke.
My Gemini invoice tests started with one action per turn, as I wrote Saturday. Median: 266.9 seconds per saved bill.
Later runs batched actions and stopped at Saved. The median fell to 46 seconds.
Then I checked the saves. 7 of 10 runs saved a bill, down from 8.
The faster runs failed in new ways.
Two returned an empty turn partway through the form. One hit Google's safety filter.
Two fixes went in. An empty turn now gets a nudge. A safety block now logs as blocked_by_model, not as an API error.
invoice-cua-probe status: public since Nov 10.
Before publishing, the export got checked. A fresh install passed 31 tests. gitleaks and trufflehog found no leaks.
Run reports stay out of the repo. The Gemini batch alone was 24 screenshots.
Measured results are in the README.
@kimmmmmhaha Agreed, it belongs in the same config as the model ID. Both decide what the call costs.
I'd also fail or log any call that goes out without an explicit level. A silent default is the part nobody notices until the bill does.
Gemini 4 Hydrogen vs 3.8 Flash, the model my invoice agent runs, on one computer-use step.
The step: read a mock invoice form screenshot, pick the first action.
Hydrogen at medium: 6.4 s. Flash at low: 12.3 s.
Both passed, so the pass check didn't separate them. Tokens did.
@Prospernwo Pinned. In this test I set the level explicitly on every call and ran medium against high. Medium was enough for this step.
Going forward I'd pin it per step type. Letting each step choose, or leaving High as the default, is how the token bill jumps.
@hsn__slm Making the count visible in CI is the useful part. Otherwise the test-file exception grows quietly.
I wouldn't auto-promote at a fixed number, though. I'd keep the tag as a signal and the move as a decision someone makes.
Caveat: one run per level, one step, and every run's first move was the same click into the Payee field. Costs are estimates from token counts and Google's published prices, not a bill.
The swap needed two changes: the model ID and an explicit thinking level. Medium was enough for this step.
Per Google, Hydrogen shipped Nov 12. High thinking is the default, so set the level yourself.
Thin test: one clean step, first action only, one call per level.