If you're so confident your test ideas are gonna win, you're not testing strategically enough.
The most game-changing outcomes I've seen came from tests that surprised people. With real disagreement before the hypothesis, and real surprise after seeing the numbers.
If I have learned something from conducting +500 rigorous tests across ecommerce sites, it's that the teams that get the most out of Growth Experimentation aren't testing to prove they are right.
They test as if they're wrong.
The idea that A/B testing hurts your paid media performance keeps circulating in ecommerce.
This deserves a proper debate, not just social media noise.
That's why we're hosting a live debate on this with two sides: people who believe testing genuinely affects CAC, and people who think that's a misread of noisy data.
Who would you put on the panel? With real data over opinions.
Drop a name and which side they'd defend.
The latest Live with @Intelligems session with @shanetroach was so insightful. His hot take: You're leaving money on the table by showing the same homepage to new and returning visitors.
Worth a watch ๐
One interesting observation from #CircusLondon last week: "CRO" barely came up.
Not among people running hundreds of experiments a year at some of the biggest ecomm and consumer brands in the world.
Over the years I've been attending this conference, the conversation has clearly shifted from WHAT to test to HOW to test โ at scale, rigorously, with AI in the mix.ย
And those people don't call that CRO ("Conversion Rate Optimization"). A few still carry it in their job title, but it's mostly the exception. I've even seen large companies hiring for experimentation roles that screen out CRO talent, just because they don't get the connection ๐คฏ
The fascinating part is that the principles, skills, and methodologies are largely the same. CRO specialists are in a privileged position to drive innovation and business growth well beyond moving pixels around on a screen.
The opportunity is there. Most just haven't claimed it yet.
@JCGiusto runs experimentation at @mengotomars. His tagline: "I scale eCom's with experiments and a P&L, not vibes."
We talked about what that actually means in practice at the latest "Live with @Intelligems ๐".
Worth a watch if you care about actual financial ecomm growth.
Just wrapped a Live with @Intelligems debrief of #CircusLondon with @paulrandall from Speero. Two days, 280 practitioners, and some of the most honest conversations I've heard about where AI is actually landing in experimentation workflows.
If you're thinking about where this is all heading, this one's worth 30 minutes. Recording here ๐
We used to joke that one person couldn't run an entire CRO program. With AI, that joke is aging fast.
First Live with @Intelligems with @VictorPayCRO. 30 min on how the workflow has actually changed. Replay ๐
Few verticals give you a feedback signal as honest as ecommerce.
Not a click. Not a form fill. Not a free trial. A purchase. People don't part with money unless they mean it.
That's what makes experimentation here different. You're not measuring interest. You're measuring behavior with skin in the game. When a variant wins, it earned it.
Most other verticals are trying to approximate that signal. Ecomm has it by default.
If you're sitting on that signal and not running controlled experiments to back your decisions, I'm not sure what you're waiting for.