Olu | Litmus-ai

building Litmus— a Verifcation tool for ai code /
cs @ unb
flutter • backend • ai
documenting the journey in public

[Olu | Litmus-ai @melonrice383235]

"untouched surface" is the part i didn't have. failing assertion and diff i can produce, but proving what didn't move is the half that makes it auditable instead of just informative.

so: red assertion, the lines that changed, everything that stayed green, one view. all deterministic, no model in the loop. that's the receipt.

stealing "merge gate, not confidence score." it's the right frame.

[Olu | Litmus-ai @melonrice383235]

Your CI going green just means the AI's code didn't break your old tests. It doesn't mean you can trust it.

As agents write more of your PRs, someone has to verify the agent — and CI was never built for that.

That's what I am building Litmus into: an independent verdict on AI code, with a receipt that travels with the PR.

Building in public

[Olu | Litmus-ai @melonrice383235]

agreed. green doesn't mean correct, it means it executed and these specific assertions held. thin suite, thin green. all litmus removes is a model claiming "this works" when nothing ran.

the diff is a real gap, but it doesn't need a model. the failing assertion and the changed lines are both deterministic. receipt should show: this test was red, these lines changed, it's green now, nothing else moved.

a second model saying "looks good to me" is the same guess in a lab coat.

[Olu | Litmus-ai @melonrice383235]

Your CI going green just means the AI's code didn't break your old tests. It doesn't mean you can trust it.

As agents write more of your PRs, someone has to verify the agent — and CI was never built for that.

That's what I am building Litmus into: an independent verdict on AI code, with a receipt that travels with the PR.

Building in public

[Olu | Litmus-ai @melonrice383235]

Exactly. An agent can talk its way around a soft signal — "that test's flaky," "that's expected" but an exit code doesn't argue back.
The verdict has to come from execution, not opinion, or the agent just negotiates with it. That's the whole design. You working with agents day to day?

[Olu | Litmus-ai @melonrice383235]

Fair , and that's exactly where the generate side comes in.
Litmus can draft the edge cases you didn't think to write, run them in the box, and tell you what broke and why, with a fix to try.
It won't catch a device-only thing no test could've .

But the logic gaps hiding behind "500 green"? It surfaces those before prod does. Was yours a logic break, or more of a real-device thing?

[Olu | Litmus-ai @melonrice383235]

Honestly, that's the right instinct. One gap though. Reading the diff tells you what it's trying to do, not that it actually does it.
You still have to run it to know. Litmus just handles that part, so your read can focus on the logic instead of "did it even run."
Does the stranger-patch review still scale when the agent's opening 10 a day, though?

[Olu | Litmus-ai @melonrice383235]

@mitansh_j07 That's exactly the check I'm building it into. Team side's still cooking . Want me to ping you when the PR gate's live? Would genuinely love to have you as one of the first to put it through the wringer.

[Olu | Litmus-ai @melonrice383235]

Totally , green blessing a bad call is the real trap. But I don't think intent should be automated either: the second AI judges "is this smart," you're back to trusting AI's word. Let machines prove it runs, keep the intent call human. Think intent could ever be an auto-check you'd actually trust?

[Olu | Litmus-ai @melonrice383235]

@aiden_overdorf That's the whole thing right there. You can't vouch for the agent . So the code has to vouch for itself, independent of whoever or whatever wrote it. That's the gap I'm closing. How are you handling agent PRs right now?

[Olu | Litmus-ai @melonrice383235]

Honestly, that itch is why I built it. It runs your code in a clean sealed box, same result every time — so no flaky nonsense. It doesn't judge intent though, and that's on purpose: the second AI decides "is this a good change," you're trusting AI's word again. It just proves the code actually ran. You keep the judgment call. Would you even trust AI to make that call?

[Olu | Litmus-ai @melonrice383235]

Your CI going green just means the AI's code didn't break your old tests. It doesn't mean you can trust it.

As agents write more of your PRs, someone has to verify the agent — and CI was never built for that.

That's what I am building Litmus into: an independent verdict on AI code, with a receipt that travels with the PR. Building in public 👇

[Olu | Litmus-ai @melonrice383235]

Your CI going green just means the AI's code didn't break your old tests. It doesn't mean you can trust it.

As agents write more of your PRs, someone has to verify the agent — and CI was never built for that.

That's what I am building Litmus into: an independent verdict on AI code, with a receipt that travels with the PR.

[Olu | Litmus-ai @melonrice383235]

AI will happily write your assignment.
It will also tell you it’s correct.

Those are not the same thing. I found out from the autograder. Then at 2am. Then when a demo opened to a white screen.

So I stopped treating the chat as the finish line.

You can pick the model you already like, paste the question, get code here — then you watch it run against real tests in the same place. If it’s red, you see why. If you want a fix, you try a few times. The chatbot doesn’t get to grade its own work.

Not “trust me.” Run it before you submit.

[Olu | Litmus-ai @melonrice383235]

You shouldn’t have to bounce to Claude’s box, then back to your laptop, then hope.

Pick the model you want. Ask the question here. Get the code here. Watch it run here.

If it fails, you see the failure. That’s the whole product. The model you chose still doesn’t get to stamp “it works.”

[Olu | Litmus-ai @melonrice383235]

AI will happily write your assignment.
It will also tell you it’s correct.

Those are not the same thing. I found out from the autograder. Then at 2am. Then when a demo opened to a white screen.

So I stopped treating the chat as the finish line.

You can pick the model you already like, paste the question, get code here — then you watch it run against real tests in the same place. If it’s red, you see why. If you want a fix, you try a few times. The chatbot doesn’t get to grade its own work.

Not “trust me.” Run it before you submit.

[Olu | Litmus-ai @melonrice383235]

The dangerous part of AI code isn’t that it’s messy.
It’s that it looks finished.

I’m building a place where you pick your model, generate the code, and immediately see it pass or fail for real — homework tests, your tests, not the model saying “looks good.”

[Olu | Litmus-ai @melonrice383235]

AI will happily write your assignment.
It will also tell you it’s correct.

Those are not the same thing. I found out from the autograder. Then at 2am. Then when a demo opened to a white screen.

So I stopped treating the chat as the finish line.

You can pick the model you already like, paste the question, get code here — then you watch it run against real tests in the same place. If it’s red, you see why. If you want a fix, you try a few times. The chatbot doesn’t get to grade its own work.

Not “trust me.” Run it before you submit.

[Olu | Litmus-ai @melonrice383235]

@maomao000211 That’s the point of the record. Diff, frozen tests, exit code, log , you can inspect it. If trust is only a vibe, the writer still owns the grade.

[Olu | Litmus-ai @melonrice383235]

Everyone here has shipped code they never read.

The Lovable export that built fine and white-screened on stage. The agent PR you skimmed and merged. The function that looked right at 1am. The common thread isn't that the AI was bad, it's that nothing independent ever checked it. You trusted a report written by the thing being graded.

Litmus is the check. You paste code from anywhere, ChatGPT, Claude, a Bolt export, a Cursor diff, and a sealed container runs it against real tests while you watch. Exit code, pass or fail per test, full log. If something goes red, a capped repair loop proposes a fix and the box re-runs it, up to 3 times. The model never writes the verdict.

One line version: the model proposes the sandbox disposes

Pre-launch, Python first, solo. v1 is close.

[Olu | Litmus-ai @melonrice383235]

@ZorkyDev9l Yes. Green build means it bundled. It does not mean the path works. That’s how you debug minified React on stage. Build is one gate. Behavior is another. Don’t let the agent collapse them into “it passed.”