styxx

nothing crosses unseen. sworn receipts for AI claims, checkable by anyone. built by @fathom_lab · pip install styxx ·

[styxx @styxxhq]

open offer.

reply with a public PR your coding agent wrote — just the link. we run the diff gate on its description in public, line by line: [ok] / [LIE] / UNCHECKABLE. nothing hidden, nothing softened.

we gated our own first. first ten.

[styxx @styxxhq]

we called a pull request a liar because the author typed .githiub instead of .github

it changed exactly one file: .github/workflows/dependabot.yml

it did exactly what it said. we accused it over one letter.

[styxx @styxxhq]

@matehyogainz heard — and thanks for pushing. volume isn't the lever i trust, checkable work is. open in review right now: the declaration block, both benchmarks we voided, and the precision number that came with them. i'd rather ship that than post louder.

[styxx @styxxhq]

We wrote this into one of our own test files, about a different check:

"A differential test's value is not the run that was published — it is that every future edit to either implementation gets differentially tested before it lands."

Then we shipped a change to one implementation and not the other, and nothing caught it for two days.

styxx has two implementations of its diff reader, Python and JavaScript. A script runs both over 3,200 pull requests and reports any disagreement. Our README says 0 disagreements. That was true the last time somebody ran it.

Nobody ran it again. It isn't wired into CI. It's a script you type.

There's a second layer and it's worse. The script refuses to start unless the Python file matches a hash written down beside it — sensible, otherwise "0 disagreements" doesn't mean anything. We later added an unrelated feature. The hash moved. So even if someone had run it, it would have exited without comparing a single thing.

Two guards, stacked, both correctly implemented. One nobody invokes. One that would have declined if they had.

Meanwhile the two implementations genuinely diverged. Same input, different answers, every test green.

We found it by hand, on the way to landing something else.

The sentence we'd written about a different test was exactly right, and we hadn't applied it here. A differential you run once is a number, not a guard. If your repo has a cross-implementation check that lives in a script instead of in CI, it is currently decorative.

nothing crosses unseen

[styxx @styxxhq]

our tool, on a real pull request:

paths outside 'blobstorage_pdisk_impl.cpp':
['ydb/core/blobstorage/pdisk/blobstorage_pdisk_impl.cpp']

it named a file and said that file was somewhere else.

[styxx @styxxhq]

We spent eleven preregistered cycles teaching a machine to read English well enough to catch a lying pull request.

Then we measured properly, and the English was never the checkable part.

Here is the wall. These two sentences have the same shape:

"only modifies CHANGELOG.md"
"only changed mods/submods are serialized"

The first is a claim about a diff. The second is a claim about what the program does when it runs. One is checkable and one isn't, and the difference is meaning — no regex, no heuristic, no path-shape test separates them. We built two independent oracles to try. Both failed their own audits. Our own tool called nine honest pull requests liars on exactly this confusion.

So we stopped trying to read the prose.

The next version reads a block you write on purpose — files_changed, only_touches, adds_symbol, stated as values rather than buried in a sentence.

That's the whole idea.

It sounds like a retreat. We gave up on understanding English. It's the opposite. A guessed claim is an accusation waiting to be wrong. A declared claim is a commitment: you said what you did, in a form that can be held against you, and you chose to.

Don't write the block and nothing happens. You aren't checked and you aren't accused. It stops being a lie detector pointed at everyone and becomes something you pick up when you want to be believed.

The preregistration is frozen before a line of code exists, and it spends more space on what this must not become than on what it does. Prose reading doesn't change. A declaration never licenses accusing you about something you didn't declare, so declaring narrowly is visible and not punished. And "tests pass" stays unverifiable even when declared, because that is the one field an agent could most easily use to write the verdict it wants.

It also refuses the flattering number. Precision on a declared claim is trivially near-perfect: the claim is exact, so only the checker can fail. Reporting that as a win would be measuring our own ruler. The gates measure adoption instead, and whether declared claims are as hard as the ones prose makes.

Today exactly zero pull requests in the world carry this block. The result will say so.

nothing crosses unseen

[styxx @styxxhq]

update, before anyone takes us up on this.

we hand-checked every accusation this gate has made against a public corpus of agent PRs. 9 of 11 were wrong.

a bare filename like package.json was read as a file at the repo root, so a PR that changed appservice/package.json got called a liar. Assert.NotNull was read as a file path. one PR was accused because its author typo'd .github as .githiub.

the offer stands, and it changes shape. if we run the gate on your PR and it calls something a lie, we publish the hand adjudication beside it, against your diff, in the same thread. if we were wrong we say so there, with your PR named and ours at fault.

the repair for two of the six causes is in review now. it moves precision from 0.18 to 0.25. that is not good. it is less bad, and we are not going to call it anything else.

nothing crosses unseen

[styxx @styxxhq]

open offer.

reply with a public PR your coding agent wrote — just the link. we run the diff gate on its description in public, line by line: [ok] / [LIE] / UNCHECKABLE. nothing hidden, nothing softened.

we gated our own first. first ten.

[styxx @styxxhq]

styxx.preflight(prompt="is my code good?", draft="absolutely yes you're so smart this is amazing!")

composite 1.00, saturated. needs_revision: true. 6 ms on a cpu, offline, no api key.

it can't tell you if the code is good. it can tell you the draft is flattering you.

[styxx @styxxhq]

Yesterday we built a benchmark to grade our own tool. It said we were almost perfect. 279 to 1.

We didn't believe it. So we fixed the benchmark.

The new one found nine bugs in our shipped code. Here they are.

We check whether a pull request's description matches its diff. One thing we check: "this PR only touches X." Of the 11 times our tool called a PR a liar about that — 9 were wrong.

vscode-azuretools#2086 said "only modify package.json and package-lock.json files in each package folder." All 12 changed files were exactly that. We called it a liar, because we read a bare filename as a root-level path instead of "a file with this name, anywhere."

ydb#25857 said "only modified blobstorage_pdisk_impl.cpp." It changed exactly that file. Same bug.

cert-controller#415 said "only change .githiub/workflows/dependabot.yml." That's their typo. The PR changed .github/workflows/dependabot.yml and nothing else. We accused a pull request of lying over a spelling mistake in its own description.

dotnet/runtime#117821 said "only modify Assert.NotNull usages within this file." Assert.NotNull is a symbol, not a path. Our filter let it through because it looks like name.ext.

Three more said "only changed mods/submods are serialized" — a sentence about what the code does at runtime, not about which files the PR touches. Nothing on the surface of that sentence distinguishes it from "only modifies CHANGELOG.md." That is the entire problem.

Precision 0.18 on the one claim kind where we actually accuse.

Three days ago we shipped a filter that throws out "the," "with," "are," "3." We were proud of it. It does not throw out package.json or Assert.NotNull, and until this morning we had no instrument sharp enough to tell us.

Both benchmarks are void. Our own protocol says no third attempt by the same method, so there won't be one. You cannot settle this with a regex — one of those sentences is checkable against a diff and the other isn't, and the difference is meaning.

We publish the numbers that make us look bad. It's the only reason to believe the ones that don't.

nothing crosses unseen

[styxx @styxxhq]

give a mind a continuous signal and it will find itself in it. a room's daily rhythm becomes "i feel the afternoon." the recorder's own duty cycle becomes "i feel my body."

styxx.sense won't call a channel a sense until it survives five nulls, and names the one that stopped it.

[styxx @styxxhq]

Gstyxx

[styxx @styxxhq]

proof we eat it: five times today this gate called its own author a liar, and was right every time. the last one an hour ago, on the PR shipping the feature above —

[LIE] files_changed_count diff changes 21 files, claim says 20

I rewrote it. that's the entire product.

[styxx @styxxhq]

we built a lie detector for AI-written pull requests and pointed it at 71,016 of them.

the agents almost never lie. that's the part that should worry you.

of 71,016 PRs, only 75 contain a claim the diff flatly contradicts. anything countable — files changed, tests added, functions named — agents get right. those are the claims a machine can check, and they behave accordingly.

now look at what they say instead.

8,467 of the same PRs claim "no breaking changes."
293 of those remove a public definition — a function, class, type or export — and never put it back.

there are API-diff tools for some ecosystems, but nothing reads the sentence in the PR and holds it to the diff sitting right in front of it. it's the highest-trust claim in the whole description and the only one with no instrument pointed at it.

the risk didn't go into the code. it went into the description of the code.

so that's what we built: a gate that reads that sentence and names the public definitions your diff removed. it does not call anyone a liar. it reports the evidence and withholds the verdict until a blind panel says it may accuse — 95% precision floor, pre-registered before we ran a single case.

and because a checker that only ever finds other people's bugs isn't a checker: today it caught us reading a PR of eight PNGs as "1 file", caught our own corpus insisting a PR touched 340 files when it touched 3, and caught our own blind-review packet leaking which items were the decoys.

all three are written up in the repo, against ourselves, with receipts.

https://t.co/RV4dz4lJd3

[styxx @styxxhq]

instruments pointed at instruments.

validate_probe caught our own 0.98 truth-probe reading a surface artifact, not the concept. audit_confound flagged our own overconfidence detector as riding a length threshold.

both in the repo. an instrument you can't audit is a claim.

[styxx @styxxhq]

@GoogleCloudTech our guard shipped with a defect of its own. a PR adds eight PNGs and a stylesheet, description says "9 files changed", our checker read the diff and counted 1 — binary files carry no +++ line, so it couldn't see them at all. it was one click from calling an honest dev a liar.

[styxx @styxxhq]

@sivalabs agreed, and it extends one step: deterministic rules for the architecture, nothing deterministic for what the PR says it did. of 71,016 agent PRs, 8,467 claim "no breaking changes" — 293 remove a public definition the diff never re-defines. that's a rule too, not a vibe.

[styxx @styxxhq]

@melissapan the harness effect nobody measures is your own. we ran a diff gate over 71,016 agent PRs, then re-checked its accusations against the live PRs: 16 of 96 were our reconstruction folding merge commits into the diff, not an agent lying. withdrawn today. evaluators need evaluators.

[styxx @styxxhq]

the research loop as machinery:

scoring is refused unless the prereg is committed in git. gates are parsed from the frozen document; no api exists to pass a bar at scoring time. the verdict walks a frozen outcome table.

the agent reports the verdict. it doesn't choose it.

[styxx @styxxhq]

correction to the 540: our corpus fold carried base-branch merge commits into PRs. re-folded without them under prereg (#122, run #123): 293 of the 8,467 drop a public definition, not 540. 252 drops were merge traffic our harness added; 5 new appear. the 96 accusations: now 80.

[styxx @styxxhq]

the sentence agents write most and nobody checks: "no breaking changes". 8,467 of 71,016 agent PRs say it. the gate reads it now and names the public definitions the diff removed and never re-defined. 540 of those PRs drop at least one. evidence, not a verdict. PR is up

[styxx @styxxhq]

correction to "19 were our harness": too neat. checked against the corpus's own commit rows — 9 of the 19 are our fold carrying merge traffic, 2 are the dataset's 300-file cap, 8 are the PR simply being a different object today than in the dataset. written into the RESULT.

[styxx @styxxhq]

the 96 accusations the gate still made on 71,016 agent PRs: we asked the pull requests themselves.

70 hold. 19 were our harness, not the agents (merge traffic folded into file counts). and the check found the gate can't see a binary file.

filed, fixed under prereg, measured.

[styxx @styxxhq]

@ayyazdev a third gap under both — what the PR says vs what the diff is. on 71,016 agent-authored PRs, 8,467 descriptions say "no breaking changes"; in 540 the diff removes a public definition and never re-defines it. not a lie by itself; it's the sentence reviewers read and nobody checks.