How come Bayesian and frequentist guarantees can be so different, even when the results are the same? In 1994, Philip Dawid published a fantastic paper about this.
Let's look at this from an experimentation perspective!
https://t.co/SJu1Lyxonh
We are building Allyon: Battlecry, the first monster trainer where creatures actually learn from you. It's closer to the Pokemon anime than the games: Allyon understand your voice, learn bespoke behaviors, have personalities and memories, and form lasting bonds with you.
1/7
In the January/February 2025 @HarvardBiz (and online now): a new article that explains the steps that companies should follow to get better at experimentation
https://t.co/koAnw1Rat1
from me, @IavorBojinov, @rameshjohari, @spschmit and Martin Tingley
Fuck it ship it.
I wrote this while kinda being annoyed by people asking for advice.
But know if that I sound angry and haughty, it’s only 40% intentional.
https://t.co/2T4vSAnMqv
@seanjtaylor And 💯 on priors are great; or rather they are awesome. Bayesian experiment analysis with "proper" priors is very compelling imo. Not leveraging them is throwing away the baby with the bathwater, which might as well be the tl;dr of the blog post 😀
@seanjtaylor Thanks for the kind words! Thinking of the MDE in a power calculation as some sort of prior is interesting -- I have always thought of them more as a "worst-case analysis" which feels very frequentist compared to the "average case analysis" of Bayesian analysis.
@teej_m I'd say this is more an issue of using ratio metrics rather than averages; it's often unclear if a ratio metric going up (on average) is good or bad. Histogram doesn't really help here (a shift to the right can still be both good and bad)
We are now accepting submissions for @CODEConference 2023!!!
We welcome any and all research related to digital experimentation, whether it comes from academia, industry, or somewhere else. If you have any questions about whether your work is a good fit, shoot me a DM/email.
@Chris_Said That p-value does not inspire too much confidence in that point estimate though; some quick math suggests that the corresponding 95% confidence interval is -50 to 260
Last couple of weeks I gave a few talks on the Sparks paper, here is the MIT recording!
The talk doesn't do justice to all the insights we have in the paper itself. Neither talk nor twitter threads are a substitute for actual reading of the 155 pages :-)
https://t.co/OF45HHpoaY