Yoni

Software Engineer

[Yoni @yorom24]

Two AI agents reviewed the same backend and recommended the same fix.
One said it would buy "10–20%". The other said "10–30%".
Neither number came from anywhere.
So I built an open-source tool that makes AI code reviewers show their evidence. Then I tested it 5 times.

[Yoni @yorom24]

My work in Sanoy24/ytgrab was selected for the XERJ Corpus Hub: AI coding agents retrieve it as a worked reference instead of guessing. https://t.co/5dgt313EdI #OpenSource #xerj @xerj_org

[Yoni @yorom24]

What I need now is people, not code: backend engineers to independently review an open-source repo, so the next test isn't graded against my own answers. Reply or DM if you're in.

And: what's the most confident wrong number an AI tool has ever given you?

[Yoni @yorom24]

Two AI agents reviewed the same backend and recommended the same fix.
One said it would buy "10–20%". The other said "10–30%".
Neither number came from anywhere.
So I built an open-source tool that makes AI code reviewers show their evidence. Then I tested it 5 times.

[Yoni @yorom24]

Try it in one command:

npx skills add https://t.co/aKxxoPjd2d

It's also a GitHub Action on the Marketplace. The code, the test rules, every raw report and every failure: https://t.co/Gilw7WZMUx

[Yoni @yorom24]

@GaelBreton Only 2🤣

[Yoni @yorom24]

I built JevPolicy an open-source TypeScript runtime that turns probabilistic AI judgments into deterministic, auditable decisions using policy-as-code.
Includes replay, shadow evaluation, semantic diffs, fallbacks, and OpenTelemetry.
https://t.co/Xv0hyYUJTa

[Yoni @yorom24]

Reviewed the same repo twice once as me, once as a fresh agent with no memory of my findings. It caught a bug I'd read right past. Built this so AI code review can't fake numbers or invent findings.
@CodeTrendr

https://t.co/Gilw7WZMUx

[Yoni @yorom24]

@ManUtd Sanoy 17

[Yoni @yorom24]

Self-preference inflates absolute scores, not relative deltas between conditions. So a Delta-over-baseline finding survives it. What does not survive: any absolute claim about tone quality on tasks the judge also authored. Two separate controls needed.

[Yoni @yorom24]

I kept treating preference leakage and self-preference bias as one problem under "cross-family rotation." They are not the same bias. They live at different pipeline layers and require different fixes.

[Yoni @yorom24]

A second path: Gemini generates the chosen preference pairs and then judges tone on the held-out partition. The model card already names this as an open gap. Diagnostic: stratify held-out scores by generation_model. If Gemini-authored tasks score higher, that gap is the measure.