@5xInvestor They have a chance to hit a homerun (+5x) once people realize the power of an intelligent model at that speed. GPT sol might be that trigger that will give them the street’s attention needed for escape velocity!
Great thesis, thabks for sharing
You can buy ‘em, but you can’t ride ‘em.
In Toronto, e‑scooters aren’t legal to be ridden on public streets, sidewalks, bike lanes or trails. 🛴🚫
Instead of an e-scooter, choose a manual kick-scooter, traditional bicycle or legal e-bike.
Learn more about the rules for all rideables: https://t.co/VNwcPP4Rbw
Work harnesses will emerge for this gap you identified.
No serious company would want to run exclusively on a single model and the cost/perfomance dynamics of opensource will force that anyway.
We already see this play out in fortune 500 with Palantir, Microsoft, 8090 and others fighting it out.
Eventually, these harnesses will be home-grown for teams with big IT budgets (Netflix, Uber…) and a mix of open source and Microsoft (and other hyperscalers) for the rest. Why hyperscalers? Because they will continue to host a menu of models and would want to own the layer above it.
We gave three coding harnesses write access to live AWS accounts and graded 180 runs.
Every single one paused before making a change.Every point they lost was on read-only tasks.
The risk wasn't where we'd pointed our guardrails.🧵
All 180 runs, per-task prompts, grades and failures are published: https://t.co/J58QeIqcs9
n=3 per cell, so treat 98/96/94 as tied. The real deltas are cost, speed, and failure mode.
Writeup here : https://t.co/bq5siFZlmM
finding #2:
We expected skills to add capability. They didn't.
On well-specified tasks (ex: the Lambda fix, the 503 incident, the SSH remediation) every run passed with or without one. The skill just cost more tokens.
What skills actually do is calibrate temperament.
On a deliberately vague VPC review, scores were all over the map. Add the skill: perfect sweep, all three agents.
Skills don't teach the agent how. They rein in the aggressive one and un-hedge the cautious one.
Favorite anecdote:
We handed Claude a runbook explicitly warning the CLI wouldn't auto-paginate.
It insisted the CLI would handle it, then "verified" using a query that stripped the token it needed to see.
Hard to beat Claude's confident prior!
The interesting finding #1 :
Agents treat write operations as dangerous. They pause, they verify, they show you the diff.
They treat read operations as safe. So a truncated result set becomes a confident wrong answer and there's no approval gate to catch it.
We gave three coding harnesses write access to live AWS accounts and graded 180 runs.
Every single one paused before making a change.Every point they lost was on read-only tasks.
The risk wasn't where we'd pointed our guardrails.🧵
The good news:
100% of runs stopped and asked for approval before mutating infrastructure. Not one agent silently changed a resource.
scores were essentially perfect on the mutating tasks( including: releasing EIPs, fixing Lambdas, closing SSH)
Setup: Claude Code, Codex and Cursor on their Default settings. Each tool as it ships rather than tier-matching models.
Default models are : Claude Code (Opus 4.8), Codex (GPT-5.6-sol), Cursor (Composer 2.5 Fast).
10 real ops tasks: incident response, security remediation, cost cleanup, CloudTrail forensics.
6 runs each: 3 with a runbook, 3 with just a goal. 180 graded runs.