A real Custyn pull request, with the risk factors shown. This one scored Medium (55/100). Custyn is running in shadow mode: it reports, but doesn't merge or block the PR.
An agent PR says it fixed the failing checkout test. The diff changes one line of code and regenerates a 400-line snapshot file. Open the snapshot diff first. If the expected output changed to match the new behavior, the test didn't get fixed, it got rewritten to pass.
@aliouftw Most of that rebase wait comes from tall stacks sitting open while main keeps moving. Keeping stacks to 2 or 3 PRs and merging the bottom one as soon as it's green cuts a lot of the conflict churn.
@asyncrohan One habit that helps with that. Read the ticket before the diff. When you already know what should ship, the plausible extra changes stand out instead of blending in.
@guptasidddharth Good call, just make sure "require review from code owners" is on in branch protection too, otherwise CODEOWNERS only asks for a review.
@GergelyOrosz CI gets hit first because every agent PR runs the full suite, even the ones that only touch docs. Running only the tests a change can actually reach buys some time before verification becomes the wall.
An agent PR adds one line to package.json for a date formatting helper. The lockfile diff is 1,800 lines and GitHub collapses it by default. Worth expanding it anyway to see how many packages came along, and whether any of them run install scripts.
@coldboothq Test first is a good call. Once catch, if the agent wrote those tests in the same PR they tend to describe the code, not the requirements. Checking each test against the ticket before reading the code can catch that early.
@livesanches@strube@X For a solo app that can be one small GitHub Action job that runs your tests on every PR. Then an agent's PR can't show green unless that job actually passes.
@livesanches@strube@X Agreed, the agent saying "tests passed" isn't proof. That's where CI earns it's keep. It runs the tests itself on every push, so the pass or fail comes from a neutral run you can open and read.
@vishwanath_tns Agreed. The tricky part is deciding what counts as "auth" or "config" in your own repo. Write those paths down once in CODEOWNERS or a short list, and the sorting happens before anyone opens the PR.
@livesanches@strube@X Asking for the file and line behind each bug cause is a good habit. It makes the agent point at something you can check instead of just telling a story. Do you have it list which tests it ran too, or do you rerun those yourself?
Flaky test, agent fix: wrap it in a retry(3) and bump the timeout to 10s. CI goes green the PR looks tiny. Worth asking what changed in the code under test. If the answer is nothing, the race is still there , it's just quieter now.
@strube@livesanches@X One middle ground for verbose diffs is to ask the agent to list each behavior change in a sentence, then read line by line wherever the diff touches something the list doesn't mention. That mismatch is where the surprises hide.
An agent PR renames isExpired() to isValid() across 14 files. It reads like a mechanical rename, so it gets a skim. But the meaning flipped, so every caller needed a ! added. 13 got one. The 14th now refreshes good tokens and keeps expired ones.
@CaptainLevi_2k One check that helps, for each new abstraction, count implementations and callers. One of each usually means it should be inlined before anyone reviews it, and the 20 lines become readable again.
@avlihachev The conflicts are the friendly part. The ones I'd hunt for in the third PR are old field names that don't conflict at all: string keys, serializers, raw SQL, fixtures. Grep for every old name before you rebase.
Say an agent PR adds a new checkout path 'behind a feature flag.' The flag is real, but its default is true, so it's on for everyone the moment it merges. Check the default value, not just that a flag exists.