Most A/B tests don't increase revenue.
They just create noise.
After working with 3000+ teams, we turned everything into a practical A/B testing blueprint on what actually moves revenue.
Free access. No e-mail gate.
Live on @ProductHunt:
https://t.co/BKMUk3nPkA
@hey_atticus Great addition. The overall split can hide a broken one: if mobile Safari loses users in one arm, other browsers can make up the gap.
Checking SRM by browser and device takes a minute and catches exactly the redirect and bot-filter bugs we mentioned.
10,100 vs 9,900 visitors: normal.
10,400 vs 9,600: something in your test is broken.
Six checks to run before you call an A/B test, including the one that's easiest to forget. https://t.co/4c3Xrg9Mx3
Two conditions, both have to be true:
The sample size you calculated before launch is reached
Itโs run for full weeks (1 minimum, 2 for B2B)
Example: a 5% page with 550 visitors a day needs ~4 weeks to detect a 20% lift. To detect a 10% lift, it needs ~4 months, which means you should test something bigger.
Tekmetric's first A/B test ran for six months and moved conversions 3%.
Maddox's took a navbar button from 0.9% to 7.2% click-through.
Both counted as wins. What seven teams tested first, and how to pick yours. https://t.co/koKRTCUrop
Website trends that need to die in 2027:
The generic "Why Choose Us?" section.
If you have to explicitly tell visitors why they should choose you, your website probably isn't doing a great job of showing them.
Same test. Same 20,000 visitors.
Frequentist: p = 0.058. Not significant. Bayesian: 97% chance the variant is better.
Both are right. Here's why, and which one to trust when you need to make a call. https://t.co/oCRzVG9KOn
Conversion went up 8%. Refunds went from 2.1% to 3.4%. Net revenue dropped.
That's what happens when an A/B test only watches one number.
What guardrail metrics are, which ones to pick for each test, and how to set the line before launch. https://t.co/OHjTL0li02