Shaojie Deng

[Shaojie Deng @alexdeng]

@KyleCSN It all depends how the coefficient in Cuped is fit. Using pooled or weighted average from two groups. Explained here in more detail. https://t.co/9j1dQ8LiyV

[Shaojie Deng @alexdeng]

@gregakespret @ronnyk If you are interested in using this in continuous monitoring/sequential analysis too, see here https://t.co/O6wkLscKWE

[Shaojie Deng @alexdeng]

@gregakespret @ronnyk It is based on this paper:
https://t.co/sF8LYps0bY
It is using Bayes Factor, but use a prior trained with historical experiment data.

[Dean Eckles @deaneckles]

When asked to draw a regression line, my experience is people draw something closer to the PCA line.

[Shaojie Deng @alexdeng]

@drsimonj @ronnyk Sure. Let me first try to explain more in an email to you.

[Shaojie Deng @alexdeng]

@ronnyk @drsimonj 2/2 since mean of covariates for treatment and control (variant and base) are the same, they will be canceled when computing the delta. Therefore in the actual computation we don't compute it at all. The adjusted metric is simply metric-theta*covariate

[Shaojie Deng @alexdeng]

@deaneckles @ronnyk @drsimonj Also cuped is not exactly regression without interaction with treatment assignment term. We still estimate two theta for treatment and control group, then use a weighted average of the two theta in adjustment. We also normally think about super population PATE.

[Shaojie Deng @alexdeng]

@deaneckles @ronnyk @drsimonj Yes I’m aware of his paper. Using same theta is finite sample unbiased (at least under Gaussian case) reason is thetahat is function of sample covariance and variance and they are independent of sample mean.

[Shaojie Deng @alexdeng]

@deaneckles @ronnyk @drsimonj It is an in sample fit but we force the same theta for treatment and control, Vs. use separate theta fit by treatment and control in sample data. The asymptotic difference between this and regression adjustment is negligible for our typical applications.

[Shaojie Deng @alexdeng]

@lukasvermeer @LizzieEardley Delta method

[Shaojie Deng @alexdeng]

@lukasvermeer @LizzieEardley Two things are mixed up here. What Lizzie pointed out is easy to tackle with details method. What @lukasvermeer is saying is denominator movement make the metric movement hard to interpret. But when session are hardly moved, using it as denominator leads to more sensitive metrics

[Harvard Business Review @HarvardBiz]

What you need to know about implementing A/B tests at your company. https://t.co/chmzDcNHCM @ronnyk

[XKCD Comic @xkcdComic]

Drone Training https://t.co/SHSARpVxlS https://t.co/DXmAb2XtLd

[Shaojie Deng @alexdeng]

@ronnyk another published misinterpretation of pvalue as posterior https://t.co/fyirlj6JAp

[Shaojie Deng @alexdeng]

@rbelo @ronnyk @deaneckles @seanjtaylor Welch test assumes normality or CLT to kick in. For highly skewed dist, equal size T vs C makes CLT kick in way faster.