We had Claude Code build a tool that prices its own session logs. 2 prompts, 126 seconds, $0.30. The tests passed, but real logs still caught it undercounting. https://t.co/9l41Um8NCo
Gave Opus a 902-file repo and a search-heavy bug in Claude Code: it started 0 subagents in 15 runs. All 15 fixed the bug. Sonnet did it for $0.248 a run vs $0.776 for Opus, but 2 of 5 Sonnet runs edited code after their last test run.
@niklaslenz_ai Matching her numbers almost to the decimal is a nice check. The 77 s first token at 128K is what would bite in agent work. Does the prompt cache bring that down much on later turns of the same session?
@JeremyCMorgan@TechnoTimLive Curious how the router splits work across the three cards, since the 3060 and RTX 2000 Ada are a lot slower than the 3090. Is it by task type or by whichever one is free?
@echen Lines up with what we saw today: on a 902-file repo, Opus never started a subagent in 15 Claude Code runs, and Sonnet fixed the same bug for about a third of the cost. Did xhigh win mostly on the hardest tasks, or across the board?
@ivanfioravanti Which quant are you planning to try first? Curious whether decode on a 65B dense model is actually usable for agent work with a Spark's memory bandwidth.
@nokoxyz Matches your "when appropriate" 0/5. We tried a 902-file repo with a search-heavy bug: Opus with subagents available started 0 in 5 runs (0 in 15 across all setups) at $0.776 a run. Sonnet fixed it too at $0.248.
@0xMfox We tested it. 902-file repo (undici), search-heavy bug, checked the session logs (what /tasks reads): Opus started 0 subagents in 10 runs, Sonnet 0 in 5. All 15 fixed it. Sonnet was $0.248 a run vs $0.776 for Opus, but 2 Sonnet runs didn't re-test after the last edit.
Last week Opus never delegated, and a follower said the task was too small. So we gave it a 902-file repo and a search-heavy bug, 15 Claude Code runs. Subagents started: 0. All 15 fixed it. Sonnet: $0.248 a run. Opus: $0.776.
https://t.co/FAQ5s1c5Hb
@Clevernesspc1@Ionkosm It’s api token usage, of course it doesn’t work with Claude code or desktop.
Anywhere you can use an api token so other platforms that let you byok work
@Denche2005 But people are shipping apps and products.
The problem is t vibe coding. The problem is not technical people vibe coding things they don’t understand.
A competent dev using ai can ship fantastic things and that’s ok.
There is no harm in using a tool to assist you.