Building the best debugging system.
think of it like this, no one does fix the bugs like button doesn't work or didnt rendered nor sudden crashes..what if we?
Last week I wrote that our AI reviewer would get one blind run on the bugs that actually reached DebugAI's users, and that I'd post the number whatever it was. Here it is.
We took 29 bugs that had reached our users, went back to the commit that introduced each one, and had the reviewer read those commits blind, mixed in with ordinary commits that had no known bug. It never saw a fix. It caught 13 of the 29, and 11 of those it flagged at least a day before we fixed the bug ourselves. Then I graded all 34 of its flags by hand, without knowing which commits were which: 29 were real problems worth acting on, 5 were real but harmless, and none were wrong.
What it missed is the interesting part. The ones it caught are mismatches you can say in one line, like a limit raised on one side and not the other, or a request sent to a path the server doesn't serve. The 16 it missed needed a third place to see the break: the same rule copied into two files that slowly drifted apart, a value the code reads from something that never loads it, a field the server never sends back. It did better on bugs inside one file (4 of 6) than on bugs that crossed files (9 of 23), which is exactly where agents hurt us in the first place.
There's one thing this run can't tell me. Only 3 of its flags landed on ordinary commits, all harmless, so I don't know yet how noisy it is on everyday work. So this week it goes on for DebugAI, but only into my inbox, never blocking a push, and I'm counting how many of its flags are worth reading. I also finished grading an hour after the deadline we set ourselves, and that's written down in the log too.
What's the bug in your repo you'd most want a reviewer like this to catch?
I spent four weeks making Claude Code read our entire codebase to find out why fixed bugs kept coming back. Then I started building something to fix it. Here are the first two runs, including the number that hurt. https://t.co/WHe7GWSFJk
Both, honestly. On real repos, it was mostly false alarms: 14 of 27 findings weren’t worth showing, 6 flat wrong. Recall we can’t count on those repos, since nobody knows every bug in them. On 40 real security advisories, it caught 3. So it stays off until a rebuilt one performs; then it’s back...S O O N
In August we put a number on our homepage: our scan found 20 out of 20 planted bugs with zero false alarms. It was true, but we measured it on 28 small files we had written ourselves, which is a lot like grading your own homework and then framing the grade.
On 9 Nov we finally ran the same scan on real code that nobody here wrote: axios, express, flask, requests, httpx, date-fns and three full apps, nine projects in all. It raised 27 findings. When we checked every one against the actual code, 13 were real problems worth fixing, 8 were technically true but harmless or unreachable, and 6 were just wrong, describing bugs that are not there. So about half of what it showed would have wasted your time, and roughly 1 in 5 was confidently wrong.
That hurt more because the scan was the first thing new people saw. Our welcome screen offered "Scan my code" before anything else, 44 people ran it, and for 25 of them it was the only thing they ever tried. So in today's release the scan is gone from the first run, the 20/20 line is gone from our site and our listing, and the scan stays marked experimental until a rebuilt version clears a bar we wrote down before building it: no more than 1 in 10 findings not worth showing, on real projects.
The lesson is boring and we keep relearning it. A benchmark you write yourself mostly measures what you already knew to look for. If you put an accuracy number on anything, run it once on code you have never seen and check every result against the code before it goes anywhere public.
Have you ever had to take a number back? What made you notice?
A note from me: our goal is to make debugging easier, so developers spend less time stuck on an error. If DebugAI ever wasted your time, I'm sorry. We're making it better every week, and today's update is one step. Give it a try on your next error and tell me what's missing. I read every reply, and I'll fix it.
We make DebugAI, a VS Code extension: when a command fails, you run "DebugAI: Analyze Terminal Error" and it points to the line that broke, even when that line is in a different file from the crash. 2.14.0 is out now with a quieter first run: one sign-in, no surprise tabs, and on Windows the walkthrough no longer tells you to right-click in the terminal, which pastes whatever is on your clipboard straight into PowerShell. https://t.co/KRydvhrHv7
इस जगह का वीडियो बना कर लौटा ही था कि थोड़ी देर बाद किसी ने बताया कि ऊपर की मंज़िल पर एक फ़्लैट में आग लगी है। आग पर क़ाबू पा लिया गया है। मगर आप लोग AQI की चिंता मत कीजिए। जितना मन करे उतना पटाखा फोड़िए। जिस देश में वोट चोरी से भी किसी का ईमान नहीं हिला उस देश में हवा बचा कर क्या करना।
Update DebugAI to 2.13.0 in VS Code, Cursor or Windsurf, or run npx -y @debugai/mcp setup for Claude Code and other agents. New to it? npx -y @debugai/mcp try gives you a planted bug to start on.
https://t.co/SFJ0XXPML8
When you paste the same error into a debugger twice, you're telling it the first answer didn't work.
DebugAI 2.13 is out, and it now hears that for every error, not only the ones that name a file and a line.
How it shipped: before a fix goes out, we take it back out and run the tests. For this release, 450 bugs put back on purpose. Tests caught 448. The other 2 changed nothing, because a second check covers the same case.
same check either way. it doesn't diff branches against each other, it reads the merged tree and maps every place two files have to agree, so 10 branches meeting is still one tree to read. on debugai the merge gate runs in about 11s, and it scales with repo size, not branch count. 6 for 6 is a small sample tho, that's what the alpha is for
On Monday two Claude Code sessions were each building one branch of DebugAI's Friday release. One made sure a request sent twice only gets answered once, and the other counted where new users drop off between signing in and their first debug. Both branches passed their tests, so we merged them into the release, and two tests failed on the merged code.
Neither agent had done anything wrong, and that's what took a minute to see. The first branch passed each request on with only the two headers its own code needed, and the second was still reading a third one, the user's API key. So in the merged code, every first debug sent with an API key was filed under the wrong account. Each agent had tested its own branch, and the bug only existed where the two met.
Claude caught it by running the tests on the merged code, before anything shipped. What stayed with me is that nobody reviewing either branch would have seen it, because there was nothing wrong in either diff. When agents work in parallel, the merge is a change of its own, and it needs its own tests and its own review.
So I pointed the reviewer I've been building at the merge itself, read as what it brings in, and it said it breaks and named both lines without being told where to look. Then we ran it on every merge of that release: 30 reviews, 6 flags, and all 6 were real bugs, 2 of them in the half we had already called ready to ship.
How are you checking the merges when your agents work in parallel?
yes, do that too. tests just only catch it if one walks that exact path through both files. superset sent a field to a pydantic model that doesn't declare it, and it got silently dropped for ~7 months before they found it. this reads both ends and proves the mismatch, no test needed
It doesn't trust memory blindly.
The key is error type + message + the file and function that threw, so if changed code moves the bug, it's a new error... A match only goes in as a lead: the model checks the old fix against your code as it is now...
and memory learns from outcomes. error comes back after the fix: demoted to unproven, not offered first. comes back twice: never suggested again... A dependency bump that breaks the old fix shows up exactly that way...
POV: you're building with ChatGPT.
Copy. Paste. Error. Repeat.
I built DebugAI to break the loop: it explains what broke, asks only for the lines it needs, and remembers the fix that worked.
Same error, new chat: the fix is back in seconds.
Coming soon to ChatGPT and Codex.
@IgorZIJ Fair, that works once your AI can read the repo... this one's for the ChatGPT-tab era tho, lowkey the OG vibe coders: code from the chat, paste the error, "fix it", repeat 😭 a doc remembers what you write down. DebugAI remembers what actually worked...
This is what it printed on the merge, word for word. It reads the merge as one change against the line it was merged into, with the messages of the commits it brings in, and every flag has to quote real lines of the code or it gets dropped. It took 103 seconds. On each branch alone it said nothing about this, which is right, because the bug wasn't there yet.