Hi, I'm Tobias. I help QA and product teams protect their experience integrity: making sure what users actually experience works, release after release.
My first question is always: "What could actually hurt us if this breaks?"
Know your risks before you ship. No fluff. 🧵
5 Fragen vor dem Go-live:
1. Welche 3 Abläufe d��rfen nie brechen?
2. Sind genau die end-to-end getestet?
3. Welche roten Tests ignorieren wir gerade?
4. Was hat sich seit dem letzten Release geändert?
5. Wie merken wir, wenn doch etwas bricht?
Welche Frage fehlt bei euch?
@mjovanovictech Following along. What's the current time, and do you already know where it goes? Every time I profiled ours, a couple of test projects were each spinning up their own database, and that was half the bill.
@addyosmani Curious how much of the checking goes to the boring flows like login and checkout vs the feature the agent just touched. In my experience the regressions hide in the code nobody asked it to change.
@prasenx The "timeout after 30s" vs actually seeing what broke is so real. Only catch is e2e runs get slow fast once an agent writes hundreds of them. Did they say anything about how they pick which journeys get the full run?
Spec-based tests check what you planned. They can't find what got built but never specified.
That's where an exploratory session earns its keep: a charter, a timebox, a person poking at the product with notes.
When did your team last run one before a release?
@filip_hric Same with testing, honestly. The senior folks know which corner of the system burned the company last time, and that's exactly where they poke. Do you think that knowledge survives once AI writes most of the code?
@frostyz Merging away 40k lines of tests in two days and keeping 85% is impressive. The untested admin controls would worry me more than the merge, that's usually where one wrong click does the most damage. Planning to cover those next?
@dash12_dev "Fail on the old code first" is such a cheap check, and it kills most fake fixes. Loosened assertions are sneakier: an exact match quietly becoming "is not null" looks like cleanup in review. Have you caught one of those in the wild?
Your CI doesn't need to run every test on every commit.
Industry research on test impact analysis: run only the tests linked to the changed code, likeliest failures first, and about 80% of broken builds show up in 1% of the test time.
What does your pipeline run first?
Mock the whole service and every test passes, because nothing real is left to fail.
I see this when the brief is "make the tests pass". Better brief: which behaviour would hurt most if it broke? Test that for real.
Do you review what agents mock?
https://t.co/zwIcWucaQ2
Me: "Claude create e2e tests for this new service"
Claude: "Jiboogling... tests created, all tests are passing"
M: "Why did you mock the entire service? Are we even testing any application code?"
C: "Good catch. The tests are degenerate. Do you want me to fix that?"
What the hell are we doing here man
Can you test in production?
Depends on the size of the crocodile.
A typo on a settings page: sure, ship it and watch.
Payments, bookings, transfers: no. By the time monitoring sees it, your customers already did.
Know where the crocodiles are before you choose.
400 test cases for one insurance pricing rule. You need about 6.
Ages 30 and 31 run the same code, so test one age per group. Start with the most common case, then change one thing per test.
A red test then points at one rule.
How many of your tests are really the same test?
@mjovanovictech Those three ticks catch more than a dozen random durations ever will. The one that bit us was time zones: expiry saved in UTC, compared in local time on one server. Do you add a DST boundary case too, or is that overkill?
Good rule. I'd add: run it in parallel and under load too. Most flakes are timing, and a quiet laptop hides them.
1 failure in 100 isn't "mostly fixed". On a high-risk flow it's a bug you haven't met in production yet.
What's your loop count?
https://t.co/n92z7Y8lQi
@DanielLockyer 225 hours a month out of 180ms is a wild number. Makes me wonder how many of those E2E runs re-check the same paths on every build. Did you also look at trimming what runs per PR, or just move the runners?
@SebastianRoehl@HabitKit Touching every screen right before your busiest weeks takes nerve, congrats. How did you check it all, clicking through on each device, or do you have screenshot tests for the layouts now?
@Josephyala Timeouts are the one that bit me. The mock answered in 2ms, the real service took 9s under load, and the retry path had never run once. Do you keep a few contract tests against the real thing?
@GergelyOrosz Servers dying the moment they come back up is the bit I'd want in the postmortem. Recovery under full load is the scenario almost nobody rehearses, since staging never gets the herd. Wonder if they ever drilled a cold start like this.
@IamAroke Seen that one up close: hundreds of green tests, then it fell over on the first real DB call. Where do you draw the line now, real DB but mocked third-party APIs?