Cindy Sridharan

[Cindy Sridharan @copyconstruct]

How to effectively debug with agents when you no longer read or understand the code

In this post, I propose a debugging framework for agents that doesn’t rely on human knowledge of code to get to the right “root cause”

The TL;DR are the screenshots.

Link to post in the reply.

[Cindy Sridharan @copyconstruct]

Writing a post about “how to effectively debug code when you no longer fully understand it”.

This is real bug I had to fix in ~9K LOC (100% AI generated).

None of the tactics I started with here helped “root cause” the issue, lol. Had to take a different approach to isolate it.

[Cindy Sridharan @copyconstruct]

@rae1101x The change in question came from a refactor.

I have a separate “parity test” suite for any and all refactoring, and those have proven very good at preventing bad regressions, both for refactors involving code reorganization, code modifications, and code deletions.

[Cindy Sridharan @copyconstruct]

5 months and many experiments later, this was totally a case of “skill issue” on my part 😅

Like with code, agents don’t magically write great tests

If your unit tests are slop (as mine were), that’s because you don’t have proper testing guidelines for your agents (as I didn’t)

[Cindy Sridharan @copyconstruct]

end-to-end testing > unit tests, in the vibecoding era.

A massive, almost entirely agent-coded refactor passed all unit and pre-merge tests but broke a critical feature.

It was only caught due to my own excessive paranoia making me run end-to-end tests before the prod deploy.

[Cindy Sridharan @copyconstruct]

@RKV2401 There’s some overlap, but I’ve also been using new ideas for testing code not covered in the testing series, like getting agents to deliberate plant bugs at every layer and see how tests behave, mutation testing etc; which are new (to me) techniques I haven’t covered before.

[Cindy Sridharan @copyconstruct]

@CharchrisSloan Integration tests are great, but they don’t replace unit tests; they come later in the process, whereas bugs that can be caught by a good unit test should be. It allows for speedier iteration.

As for the guidelines, I’ll try to make my Claude config public, including testing.

[Cindy Sridharan @copyconstruct]

We’ll still need end to end tests, and frankly, we’re going to need all the forms of testing I’ve been talking about for a decade.

But “unit tests are now slop” doesn’t have to be the foregone conclusion.

Unit tests can be very useful even if a human will never again read them.

[Cindy Sridharan @copyconstruct]

Updated my testing taxonomy to include formal verification. Some book recs too.

Model checking, fuzzing, property based testing, sanitizers and static analysis are some of the most powerful tools one can use in the "pre-production" phase.

Also, are "types" a form of testing? 🤔

[Cindy Sridharan @copyconstruct]

@treatyoma Truly, she’s one of the vilest. Just astonishingly and unfailingly awful.

She admits she gets a torrent of racist abuse, but engages in it herself. Despicable piece of shit.

She and Indian_Bronson, both cut from the same cloth.

[Cindy Sridharan @copyconstruct]

@guraya_parneet I don’t know if I’m the best person to answer this. Build new stuff. There’s still a world of software that’s yet to be built. Start with any tool you use, and see how you can make it better. Ideas are cheap to try out. The more you do, the better you get at picking better ideas.

[Cindy Sridharan @copyconstruct]

Unfinished coding side projects are now way easier to complete than unfinished blog posts.

I’m perfectly OK with AI writing my code, even the docs.

But I flinch at the idea of AI writing my blog posts, which feels like an egregious violation of a fundamental ethnical contract.

[Cindy Sridharan @copyconstruct]

@vegardstikbakke lol, I refuse to pay for this app, so typos shall remain.

It’s also how you know I’m rawdogging every tweet myself, rather than get AI to write it (which I learnt is a thing, esp with ultra long tweets some influencers and execs post)

[Cindy Sridharan @copyconstruct]

@unsatchmo Ideally the folks building software factories or harnesses should be evaluating how best to get agents to debug, because using Claude code out of the box isn’t going to cut it.

[Cindy Sridharan @copyconstruct]

I’m literally that weirdo who wrote (all by hand) a 15K word blog post on:

- why one cant treat models as oracles
- how models gaslight you all the time
- what it takes to build confidence in the model’s output

https://t.co/SmnEGkhJzX

So yeah sure, I believe in magical woo-woo

[Dmitrii @dmitriid]

@copyconstruct I don't expect models to do things well automatically. I'm just pointing out that you believe in the magic too much. Including things "I gave it things to read" lol

[Cindy Sridharan @copyconstruct]

@unsatchmo All super valid criticisms. I’m not a researcher or on a team at work that’s tasked with building better debugging primitives using evals, testing different mechanisms etc.

I’m just an infrastructure engineer who has access to claude, and am chronicling how I use it to debug.

[Cindy Sridharan @copyconstruct]

@rae1101x That should be covered by the data integrity axis.

How data integrity is handled for stateful services which rely on replicated, partitioned data under different scenarios should answer most questions.

[Cindy Sridharan @copyconstruct]

I started this series on testing nearly a decade ago, but never published the final part https://t.co/08IA9quP9C

There’ve been so many great advances in the storage space in the ~7y I feel this is worth completing.

Have a draft from years ago I hope I can wrap up in early 2027.

[Cindy Sridharan @copyconstruct]

@rae1101x There are probably more patterns, but I need to go read more papers lol. Doing it will take time, hence the early 2027 (more like Feb or March) goal.

[Cindy Sridharan @copyconstruct]

@rae1101x Lot of it holds up pretty well, actually.

The main changes in the last ~7 years have been in:

- move towards more flexible consistency
- use of replicated logs for all manner of things
- move away from strict consensus and synchronous replication
- control plane design choices

[Cindy Sridharan @copyconstruct]

@dcarter_js Ok, so now I guess I have to do this lol.

Will try to get this done on Sunday, but if not, will have to wait until next weekend.

Might even blog, explaining where some of these ideas came from (specific senior devs I worked with, conf talks of the past, oss projects, etc)

[Cindy Sridharan @copyconstruct]

To create my claude.md, I had it read:

- my favorite programming books + blogs
- the best oss projects
- the code of the engineers I respect the most

Claude (when given proper instructions, constraints, and context) now writes better code than me, almost always.

[zack @zack_overflow]

the coding agent haters are getting dunning-kruger'd so hard right now

do you really think that your engineering ability is better than the models at this point?

it is very very likely that fable is better than you in every branch of software

it has been trained on all the code of the internet, it knows almost every trick in the book, all the design patterns of every domain, every single low-level detail, every bit-twiddling hack, it's read every single blog post from every cracked engineer that ever existed

do you really think you can beat it? do you really think you can be more creative than it?

unless you are 99th percentile expert in some niche domain of software, it simply is not possible. sorry.

[Cindy Sridharan @copyconstruct]

@dcarter_js Might need to scrub some work specific stuff, but about 95% of it is just good general software engineering principles I’ve collected from the best sources I know (if I do say so myself).

I’ll try to make it public over the weekend.

[Cindy Sridharan @copyconstruct]

@pantasw2 @manasque It has both open and closed source test suites. SQLite is a library with multiple layers; all but one of the layers has oss test suites, iirc.

There’s plenty of great ideas in the open source test suites, covering a host of domains.

[Cindy Sridharan @copyconstruct]

@dmitriid If that was your takeaway from that article, there’s no point engaging with you in good faith anymore. 👋

[Cindy Sridharan @copyconstruct]

@dmitriid None of this is true IME.

People who expect models to automatically do the right thing are bound to be disappointed. Give it good instructions, constraints, context and multiply verification by tenfold, and you get way better results than any human can achieve on their own.