@dafi2471 False HIGHs are the number I'd want before allowing any action. Do you score the proposed fix separately from the diagnosis? A plausible cause still doesn't make a rollback safe.
@jenish1235@_impl I don't know the right contact, but your eval setup sounds useful. Does it include a dependency outage right after a deploy, where the correct action is to leave the release alone?
@xazaans Calls per completed job would make a useful release gate here. Compare it with the old version on a small batch; a job can pass every functional test and still hammer production.
@theo Writing tests that find bugs. The agent tests the code as written, so a bug just becomes the expected value. We planted 12 bugs in Spring Petclinic: 100% coverage, 5 caught. With unit tests skills, 11.
@GergelyOrosz Agree, and the keyboard analogy goes further. Touch typing never made anyone an engineer, it just removed friction. The teams I see winning still think like engineers. The agent just types what they decided.
@dexhorthy Agree, heavy process for small stuff feels slow. But when did speed start beating quality? In my experience the tortoise still wins: one bug in prod eats the time saved on ten fast PRs.
@GeoffreyHuntley Agree, and measure them. Lots of noise, few numbers: METR found experienced devs were 19% slower with AI while feeling 20% faster. What did your team actually measure?
@GeoffreyHuntley Agree on experimenting. But the framing is odd: a scientist solving in an hour what took years is a breakthrough, a dev doing it means layoffs. If faster devs mean fewer devs, that's a growth plan problem, not an AI one.
@KentBeck When a scientist solves in an hour what took years on paper, we call it a breakthrough. When a programmer does it, we ask who gets fired. Maybe the gap isn't in the tool, but in what we plan to build next.
@GergelyOrosz Agree, paying it off got much cheaper. But one kind now compounds faster: weak tests. Agents copy whatever test style the repo has, so bad tests breed more bad tests. Any skills you use for tests?
@ThePrimeagen Step 2 is the real work. Deciding what to test is its own skill, it's why QA engineers exist. What works for us: the agent drafts a list of test cases first, a human reviews it, then it writes the tests.
@HamelHusain@sh_reya Makes sense. We hit the same thing grading agent-written tests: coverage was a bad judge (100%, yet only 5/12 planted bugs caught). Planted bugs became our "human labels". Do you see similar gaps with code-grading judges?
@bcherny Agree, and #3 is the hard part. "Tests pass" proves little if the tests can't fail. We planted 12 bugs in Spring Petclinic: agent tests hit 100% coverage, caught 5. With unit tests skills, 11.
@kentcdodds Mostly agree. But agents train on what's common, not what's best. With tests that gap is huge: our agent hit 100% coverage yet caught 5/12 planted bugs. With unit tests skills: 11/12.
@trq212 Agree. Skills let you borrow the precision you don't have. We measured it on tests: plain "write tests" caught 5/12 planted bugs, same agent with unit tests skills caught 11/12. Which domains bite hardest?
@SimonPTrades Test theater is real: tests written right after the code just describe it, bugs included. A fresh session plus skills that list test cases first fixed most of it for us. 5/12 β 11/12 planted bugs caught: https://t.co/ccHXBOWNqF
@deybhayden Two things cut ours: tests written in a fresh session, and explicit rules on what's worth testing. We load the rules as unit tests skills, so the agent writes fewer, sharper tests: https://t.co/mGPRCRtWCo
@AzamIntikhab 1,962 green tests, zero working screens. Tests that mock the API only prove the code matches the mock. Coverage says the same. Do you give your agents any skills for unit/integration/e2e tests?