A/B testing assumes the user opens something. A page. Pixels you can rearrange.
That assumption is breaking. Agents mediate. No DOM. No session to randomize.
A/B was always one method inside a bigger thing. What do you test when there's no UI?
Not saying A/B is dead. The human web still holds most of the revenue. 100% A/B testing in 2026 is out of position, 0% is too. It's a portfolio, and when you can't run a clean test you climb confidence with cheaper proxies (simulation, research, narrow trials).
For a decade we tested artifacts. Buttons. Headlines. Pages.
AI collapsed the cost of producing those to roughly zero. The unit of test moved up: models that generate variants.
Marcel Toben: "AI collapsed the cost of change. It did not collapse the cost of being wrong."
So the bottleneck moved from "how do we ship more" to "how fast do our beliefs actually update from real signal." That's a program-level question. No single CRO or product team owns it in the old org model. IMO that's the growth function's job.
We're prototyping this at Speero now. If you've been burned by an AI form filler that just made worse forms, where did it fall over?
https://t.co/vVGyk62CgA
A PM types "test a progress bar on onboarding" into a form. The system already knows the adjacent test was flat, the audience overlaps a live pricing test, and activation isn't this quarter's goal.
The form tells them none of it.
Intake shouldn't capture. It should interview.
It takes 3 ingredients. Context (strategy, metrics, past tests), methodology (what good looks like in your shop), and connection (it writes to the same Airtable/Jira/ClickUp the program runs on). In Claude that's Projects, Skills, and MCP. Copilot and Glean, same recipe.
I'm biased toward velocity, so push on it. Have you ever written down what a false positive costs you, in dollars, vs what a missed winner costs you?
Full piece: https://t.co/Al7T5WzLI5
There are two kinds of experimenters. One is doing science. The other is growing a business.
Same tools. Same p-values. Not the same job.
p < 0.05 is a convention we inherited, not a law we discovered. The threshold is a business decision.
Which one are you?
But the strict crowd has a real point. NIH heart trials: 57% reported a win before pre-registration, 8% after. What fixed it wasn't a smaller p-value, it was calling your shot up front. So ask before the test: what decision am I making, and what does being wrong cost?