John Deke

Senior Fellow, @MathematicaNow
Ph.D., Economics, University of Illinois at Urbana-Champaign
B.S., Economics and Mathematics, University of Kansas

[John Deke @JohnDeke]

Good science requires self-evaluation, not just evaluating others. In this post, the Director of IES acknowledges problems with how statistical significance has been used in the past and helps advance the conversation about how we can do better.

[John Deke @JohnDeke]

This is a really great blog post by the Director of IES with input from my colleague Brian Gill. https://t.co/0jX3PHL3sM

[John Deke @JohnDeke]

Haven't loved Threads; now going to try blauer Himmel

[John Deke @JohnDeke]

To whom it may concern... As a hedge, I have created a Threads account under the user name JohnGDeke.

[John Deke @JohnDeke]

@PhillipsPOBrien Interested!

[John Deke @JohnDeke]

@PhillipsPOBrien Would love a code, thanks!

[John Deke @JohnDeke]

Some friendly advice for the dedicated professionals at OMB and federal statistical agencies: https://t.co/CzjI4WnLdw

[John Deke @JohnDeke]

To whom it may concern... As a hedge, I have created a Threads account under the user name JohnGDeke.

[John Deke @JohnDeke]

Today a colleague needed a citation to support using the linear probability model in the context of estimating impacts on a binary variable in an RCT. In case others have this R&R inspired need, here ya go: https://t.co/eb3XD91zGO

[John Deke @JohnDeke]

4. Continue to interpret probability in terms of relative frequencies, not beliefs. Pull all these recommendations together and you get BASIE https://t.co/5nSrrmfIAn

[John Deke @JohnDeke]

I greatly appreciate the opportunity to write this commentary for JREE: https://t.co/Y0cARP5YjL Here's a quick summary of my recommendations.

[John Deke @JohnDeke]

3. Continue reporting the probability distribution of estimates, but do it for the right reason. We need to be transparent about the characteristics of the data we collect and analyze. Traditional (non-Bayesian) inference is good for that.

[John Deke @JohnDeke]

Should we use the population p*(1-p) or the sample-specific one? For example, say we have an impact estimate on "owns a car" and that our sample has a much lower prevalence rate than a reference population. Which p*(1-p) should we use?

[John Deke @JohnDeke]

Follow-up question: suppose we are talking about a dichotomous variable that arguably does not arise from a cutoff on an underlying latent continuous variable. One way to calculate the effect size is to divide am impact estimate by sqrt(p*(1-p)).

[John Deke @JohnDeke]

Question for meta-analysts: as a general rule, would you rather standardize by a population standard deviation or a sample standard deviation? Relatedly -- when comparing subgroups, would you standardize by subgroup standard deviation or full population standard deviation?

[John Deke @JohnDeke]

@jepusto @EduGlaze That makes sense — so in a sense, subgroup A is playing the role of the population.

[John Deke @JohnDeke]

Question for meta-analysts: as a general rule, would you rather standardize by a population standard deviation or a sample standard deviation? Relatedly -- when comparing subgroups, would you standardize by subgroup standard deviation or full population standard deviation?

[John Deke @JohnDeke]

@jepusto @EduGlaze This is all in line with my inclinations. I know I asked 'in general', but can you think of any non-trivial special case in which you'd want to use subgroup SDs rather than population? (I can't, but others might be more creative...)

[John Deke @JohnDeke]

@EduGlaze Thanks, Steve -- good to have confirmation of my priors on this :-)

[John Deke @JohnDeke]

@beckamaynard Thanks, Becka -- I agree! I have sometimes found that, because we do so rarely have that option, folks reflexively 'standardize' by sample (or subgroup) standard deviations out of habit, even when they do have a better option. A form of learned helplessness, perhaps.

[John Deke @JohnDeke]

(I have my own strong inclinations, but I'd like to hear others views)