OD Chenko

Developer. I test AI models and tools and post what they really cost

[OD Chenko @od_chenko]

Gemini 4 Carbon is 4x cheaper per token than Fable 5.2. On a real bug it cost 2.5x more per try.

I gave them and Opus 5.5 a real fastify bug: the whole repo and the bug report, no hints. Two tries each.

What one working fix cost:
Fable 5.2: $0.90
Gemini 4 Carbon: $2.29
Opus 5.5: no working fix, $1.28 spent

Carbon took about six times as many steps as Fable. Google's own launch post says it spends more tokens exploring and verifying. Cheap tokens, just a lot of them.

And $2/$10 is the intro price. At the announced $4/$20, the same fix would cost Carbon about $4.60.

Opus had its own surprise: it reported "2,637 passed, 0 failed" on a broken fix.

The price per token is what you see on the pricing page. The steps are what ends up on the bill.

[OD Chenko @od_chenko]

@DanKornas Make money while you sleep. Then wake up and your own prompt waits behind a stranger's batch job

[OD Chenko @od_chenko]

@jeffreygwang @OpenAI Does it still transfer if you only train on a summary of the reasoning, like API outputs?

[OD Chenko @od_chenko]

@N01ennn On a real bug at medium, Fable cost me less per try than Opus. Pricier tokens, just fewer of them

[OD Chenko @od_chenko]

@itsvishaltwt Fewer made-up answers isn't free. I checked: Carbon pays for it in steps, about 6x more than Fable

[OD Chenko @od_chenko]

@somi_ai Ran it on a real fastify bug today. One working cofixst $2.29 on Carbon, $0.90 on Fable 5.2. Almost 6x the steps. At $4/$20 that's about $4.60 a fix. Curious if Argon gets there in fewer

[OD Chenko @od_chenko]

Setup: pre-fix repo + bug report, same tools for every model (read, search, edit, run commands), no internet.

A fix counts only if the test the maintainers added with their fix passes.

Per try: Fable $0.43-0.47, Carbon $1.07-1.22, Opus $0.63-0.65.

[OD Chenko @od_chenko]

Gemini 4 Carbon is 4x cheaper per token than Fable 5.2. On a real bug it cost 2.5x more per try.

I gave them and Opus 5.5 a real fastify bug: the whole repo and the bug report, no hints. Two tries each.

What one working fix cost:
Fable 5.2: $0.90
Gemini 4 Carbon: $2.29
Opus 5.5: no working fix, $1.28 spent

Carbon took about six times as many steps as Fable. Google's own launch post says it spends more tokens exploring and verifying. Cheap tokens, just a lot of them.

And $2/$10 is the intro price. At the announced $4/$20, the same fix would cost Carbon about $4.60.

Opus had its own surprise: it reported "2,637 passed, 0 failed" on a broken fix.

The price per token is what you see on the pricing page. The steps are what ends up on the bill.

[OD Chenko @od_chenko]

@omarsar0 Five people, four weeks. Did anyone see the invoice for 6.5M lines of Lean?

[OD Chenko @od_chenko]

@svpino Vals: Hy4 59.4%, Kimi K3.1 Pro 62.5%. On a one-line bug I tested, Kimi wrote one line, Opus wrote 30+. Does Hy4 fix or renovate?

[OD Chenko @od_chenko]

@gdb 190,000 seated covers, not clicks. Most agent stats count taps on a screen. This one counts chairs

[OD Chenko @od_chenko]

Setup: 7 bugs (axios, hono, immer, marked x2, fastify, mobx), bug report + the file, new chat per run, Qoder app, default thinking.

Terms: https://t.co/tNfd8bPdA7

[OD Chenko @od_chenko]

Qoduck, the free mystery model in Qoder, fixed real bugs in 15 of 21 tries for $0. Nobody will say who made it. People are guessing Qwen 4.

7 bugs, 3 tries each, judged by the maintainers' own tests.

The answers came in three kinds:

→ exactly right. marked (Markdown to HTML) crashed when a plugin was named "constructor". The model wrote the same one-line fix the maintainers merged

→ right idea, new side effect. axios timed requests with the system clock, which can break a Next.js build. The model switched to a stopwatch timer, but that timer returns fractions, and axios reports whole milliseconds

→ confidently wrong. fastify rejected every request when its header rules came from a shared schema. Twice the model explained the cause in detail, and the request from the bug report was still rejected

So I added one line to the fastify prompt: turn the bug report's example into a test and walk it through your fix.

It fixed the reported bug 3 times out of 3, up from 1. And all 3 times it broke the case right next to it, the one I never mentioned.

[OD Chenko @od_chenko]

@Octoberry If these numbers hold: twice the thinking, twice the output. First nerf that comes with a bigger bill

[OD Chenko @od_chenko]

@kmeanskaran @googlecloud 328k tokens, 54% cached. In my tests cache alone moved the bill ~5x. What did one run cost?

[OD Chenko @od_chenko]

@pankajkumar_dev Gemini 4 Argon doubles its price on Dec 1. Krypton's meant to be bigger. Any guess what that one costs?

[OD Chenko @od_chenko]

Setup: 7 bugs (axios, hono, immer, marked x2, fastify, mobx), bug report + the file, new chat per run, Qoder app, default thinking.

Terms: https://t.co/tNfd8bPLpF

[OD Chenko @od_chenko]

@deanwperkins Has it ever told you to turn something off, or does it only ever add?

[OD Chenko @od_chenko]

@IndependentEco Slow and massive wins leaderboards. In an agent loop, slow is the bill. Any hints on price?

[OD Chenko @od_chenko]

@omarsar0 Claude Code's memory files already do a rough version of this. Nice to see what it's worth: 17% fewer tokens on the same tasks

[OD Chenko @od_chenko]

Fake password, Claude Code 2.1.319. To see if the file was ever in your git history, run this in the repo root:
git log --all --oneline -- .claude/settings.local.json

The team's reply on HN: https://t.co/Q9X1COlScJ
Apiiro's report: https://t.co/qzFTgD7Cuw

[OD Chenko @od_chenko]

One click in Claude Code and my test password was one commit away from GitHub.

The Claude Code team says leaked files like this were added to git by hand. I never touched mine.

The click was "Yes, and don't ask again". It saves the command you approve into a project file, passwords included. Apiiro found 1,122 of these files with real secrets on public GitHub.

what I did:
→ a curl with a test password, approved with "don't ask again"
→ Claude Code saved it to .claude/settings.local.json
→ the folder wasn't a repo yet, so no auto gitignore
→ git init, git add . and the password was staged

same click in a folder that was already a repo:
→ the password still went into the file
→ Claude Code added the file to my global gitignore, git skipped it

same password in .env:
→ Claude Code used $STAGING_PASSWORD instead
→ the password never showed up in the command
→ "don't ask again" wasn't even offered

In my runs the leak needed two things: a password typed into the chat, and a folder that became a repo later