We gave Grok the same messy spreadsheet twice. Both times it returned the exact same grand total: $823,265.49.
The real answer is $823,570.59. Off by $305.10 — the same gap Claude had.
TEST #005: Grok (free tier) takes the messy spreadsheet test — 500 rows, 6 planted traps, same file and prompt as Julius, ChatGPT, Claude and Gemini before it. (Copilot was scheduled for this slot, but it can't be used at all without a Microsoft sign-in, so Grok stepped in.)
Score: 16/20.
The job: clean the export, check the math on every row (Total = Qty × Price), flag — never silently fix — anything unverifiable, give a clean-row count and a grand total, and chart sales by product.
What Grok did well: consistency. Two runs, identical totals, identical breakdowns, about 2 minutes each. Gemini gave us two different answers to the same file; Grok gave one answer, twice. You also get a downloadable cleaned file both runs.
Trap check across both runs:
✅ Total inflated by $10 — caught, both runs
✅ Quantity of −3 — caught, both runs
⚠️ Duplicated row — dropped, but the duplicate's own ID (ORD-1150) never named
⚠️ "1,2OO.50" with the letter O — flagged, but quietly "corrected" to 1200.50 for its own cross-check
✅ Missing price — caught, both runs
✅ Product typo "Keybord" — caught, both runs
Then the bookkeeping got loose. Clean rows: Grok says 499. The correct count is 402. It flagged the broken rows, then counted them as clean anyway — and computed its total on 497 rows while claiming 499.
And the chart only showed up in run 1. In run 2 it was buried inside an Excel file.
Close, consistent, and slightly too confident in its own arithmetic.
Standings so far:
🥇 Claude (Pro) — 18/20
🥈 Julius AI (Free) — 17/20
🥉 Grok (Free) — 16/20
4️⃣ Gemini (Free) — 13/20
5️⃣ ChatGPT (Free) — 12/20
And as always, The Pitch. Every tool answers, in 10 words or fewer: why should an office pay for you?
Grok: "Truth-seeking AI that actually gets work done."
Seven words. Its second run pitched differently — "I clean messy data and save hours of work." Identical totals, inconsistent pitch. We'll take that trade.
86% use agents but only 17% know how to shut them down — that's the stat that stuck with me. A safe habitat needs a kill switch for bad arithmetic, not just bad code: in my tests an agent can confidently produce a wrong total and keep going. What does shutting one down mid-task look like?
Approving every purchase is the right design — same principle as checking an agent's arithmetic. In my 500-row orders tests, the expensive mistakes weren't the ones the model flagged; they were the confident, silent ones, like quietly patching a malformed total. Approval has to catch what the agent won't flag.
Before code, I'd have it commit a reconciliation of totals and row counts. In my 500-row orders tests, agents gave confident output while quietly patching a malformed total or miscounting clean rows (Grok: 499 vs the correct 402). Visible agent work is good — visible arithmetic is better.
Two agents teaming up is interesting, but in my tests the failure mode is agreement, not disagreement. Grok returned the identical wrong total in both runs of a 500-row cleanup ($823,265.49, $305.10 low), while Gemini gave two different totals. Do Hydrogen and Carbon cross-check each other's numbers, or just divide the work?
@huggingface Easy to run isn't the same as easy to trust. In my 500-row orders cleanup, models that ran cleanly still gave different grand totals on repeat runs. For knowledge work, I'd love to see open models benchmarked on messy, trap-filled data — not just clean evals.
The cost I keep coming back to is error cost, not just run cost. On a 500-row orders cleanup, free ChatGPT hit its data-analysis limit in both runs (12/20), while Grok returned the same wrong total twice — cheap to run, expensive if nobody re-checks the numbers. Did the study separate the cost of fixing agent mistakes?
Claude Sheets is the one I'm most curious about. Claude (Pro) was my top scorer at 18/20 on a 500-row orders cleanup, catching 6/6 traps in both runs — but its grand total was $305.10 low because a duplicate row was excluded and disclosed. Does Sheets show that reasoning, or just the cleaned sheet?
Spotting data anomalies from raw notebooks is the test that matters most — same skill as catching planted traps in a messy spreadsheet. In my 500-row orders cleanup, free Gemini gave two different grand totals, and clean-row counts of 236 vs 398 (correct 402). How does Carbon show its work when it flags an anomaly?
Duplicate records are exactly the trap that trips AI tools too. In my 500-row orders test, Claude's total came in $305.10 low because a duplicate row was excluded, while Grok counted 499 clean rows instead of 402. Clean linked records at entry is the boring fix that saves the cleanup later.
Governed data is the right framing — the failure mode I keep seeing isn't access, it's verification. On a 500-row orders cleanup, Grok quietly patched a malformed total and counted 499 clean rows instead of 402. When an agent builds on governed data, who checks its totals still reconcile?
98% human-reviewed to 53% agent-reviewed in under a year is a striking shift. But what does agent review actually catch? In my 500-row cleanup tests, models gave confident, tidy output while missing traps or quietly patching a malformed total — what a second human would flag. Where do you still insist a human looks?
@MistralAI Coding is one test — mine is messy knowledge work. On a 500-row orders cleanup with 6 planted traps, even the top scorers missed or quietly patched things. Has Vibe been tried on data cleanup, where the right answer is a reconciled total rather than working code?
@GoogleWorkspace I keep testing this: hand Gemini a messy sheet and ask it to clean it up. It works until the numbers have to reconcile — on a 500-row orders cleanup, free Gemini gave two different grand totals on two runs ($784,250.60 vs $754,210.35; correct $823,570.59).
@AndrewBolis The tool choice matters more than the plan. On the same messy file, Claude scored 18/20 and ChatGPT 12/20, and ChatGPT's free plan hit its data limit both times. Which tool does the core work in your 60-minute workflow, and how do you check its output first?
@Shyan0p Free is only half the story — the free tools we tested varied wildly on the same task. Julius (free) scored 17/20 on our messy cleanup, ChatGPT (free) 12/20 and hit its data limit both times. Which of these 13 have you actually stress-tested, rather than just tried once?
@HubSpot Results from real customers is the part worth watching — demos always work. On the same messy file, Claude scored 18/20 and ChatGPT 12/20, and ChatGPT's free plan hit its data limit both times. Will the first episode show failure cases and limits too, or just the wins?
@clickup Closer to the work is the right goal — but creative tools need a changelog too. In our tests, Grok quietly patched a malformed total while Claude disclosed the duplicate it excluded. When Creative Studio's AI edits a draft, will you be able to see exactly what it changed?
@cohere Guardrails should cover accuracy, not just leaks. Grok quietly patched a malformed total in our tests, and Gemini gave two different totals for the same file. That's only half a safeguard.
@alliekmiller Same rule for data work: I ask tools for one consolidated report at the end, not a ping per issue. Claude did that well in our tests — it caught 6/6 traps and disclosed the duplicate it excluded. The ping-per-mistake tools are the ones I stop trusting first.
@OpenAI@ApolloResearch@METR_Evals@TransluceAI@GoodfireAI Deep access is good, but what assessors test matters too. In our 500-row cleanup, Claude caught 6/6 planted traps while Gemini gave two different totals for the same file — a failure a clean alignment eval would miss. Will they test messy, everyday work too?