Raphael Merz

psych student @ruhrunibochum | interested in all things #metascience, #openscience and #rstats

[Raphael Merz @RaphaelMerz]

🚀Very excited to finally see our paper on nonsignificance misinterpretations published! 📈
Together, @ste_lee_murphy, @aurelio_fdez, Linda Reimann and I investigated the prevalence of "p > .05 = absence of an effect" interpretations. (1/4)
https://t.co/D8SlWQV21u

[Daniël Lakens @lakens]

I am hiring a PhD student on the meaningful interpretation of effect sizes as part of my VICI funded project. This is a 4 year paid position in a welcoming and collaborative environment. Find out more or apply at https://t.co/ftWYg2yxIy

[Daniël Lakens @lakens]

I think at some point we need to move from taking this as a serious ‘reason’ to be against preregistration, and call these people out for being too incompetent on research methods to take their opinion seriously. As we wrote https://t.co/OLvnDg4nU0

[Sajedeh Rasti @sajedeh_rasti]

The new workshop of Paul Meehl Graduate School, Causal Inference and Variable Control by Peder Isager, is open for registration. Check out the announcement below:
https://t.co/d0p49sKWVI

[Raphael Merz @RaphaelMerz]

@Liikennepsykol1 We don't want to sell the idea that these prevalences change in one direction or the other and I don't think we claim that there is an effect in a specific direction in the paper. I also don't have a clear-cut definition when a nonsig finding is informative. The CIs are for sure.

[Raphael Merz @RaphaelMerz]

🚀Very excited to finally see our paper on nonsignificance misinterpretations published! 📈
Together, @ste_lee_murphy, @aurelio_fdez, Linda Reimann and I investigated the prevalence of "p > .05 = absence of an effect" interpretations. (1/4)
https://t.co/D8SlWQV21u

[Raphael Merz @RaphaelMerz]

@Liikennepsykol1 @JakubTomek13 @ste_lee_murphy @aurelio_fdez In this case, we would need a larger sample to reliably estimate the true effect/change. However, the estimates and CIs are still informative in showing that these misinterpretations remain high. (2/2)

[Raphael Merz @RaphaelMerz]

@Liikennepsykol1 @JakubTomek13 @ste_lee_murphy @aurelio_fdez We don't need to debate whether the effect exists. These prevalences won't be exactly the same across years. Still, we don't have strong enough evidence to rule out that the effect is exactly zero, but you can check the CIs to see what range of values the model predicts. (1/2)

[Raphael Merz @RaphaelMerz]

@Liikennepsykol1 @JakubTomek13 @ste_lee_murphy @aurelio_fdez Thanks for the question! In the screenshot, you can replace "suggests" with "the model estimates" (see also the CIs). That said, I don't think the exact p values observed in our study are the most informative part of it. You can take these prevalences more as descriptives.

[Raphael Merz @RaphaelMerz]

📢Clearly, we have a problem. Few articles acknowledge that a nonsignificant result may simply reflect an effect that could not be found. We urge authors to reflect on their interpretations and recommend analyses with specified alternatives (e.g., minimum effect tests). (4/4)

[Raphael Merz @RaphaelMerz]

👫Of these, 23% suggested the absence of an effect at the sample level (e.g., 'groups were the same') and 58% at the population level (e.g., 'men and women are the same').

[Raphael Merz @RaphaelMerz]

@AminSinichi @ste_lee_murphy @LindaEReimann @aurelio_fdez Thanks a lot! :)

[Raphael Merz @RaphaelMerz]

Very excited to share our new preprint! 📢
We investigated how often researchers misinterpret statistically nonsignificant results as the absence of an effect. Spoiler: it’s a lot (~81% of 599 articles)! Dive into our findings and their implications here: https://t.co/8RRyiJlfEI

[Raphael Merz @RaphaelMerz]

⏳ Reminder: Sign-up for the PYMS Pre-Symposium on Dec 5th closes tomorrow! Don't miss out on this chance to connect with fellow early career researchers. 🗓️

Check out the preliminary program here: https://t.co/22za6FZ7Y1

Sign up while you still can! 👇

[Raphael Merz @RaphaelMerz]

👋Early career meta-scientists, looking to connect with others in the field? Join the Platform for Young Meta-Scientists @NLrepro Pre-Symposium on Dec 5th! A great chance to meet fellow researchers, share ideas, and build your network!🤝
Sign up here: https://t.co/T1Iu28zsMz!📈

[Raphael Merz @RaphaelMerz]

@BenCTurnbull @ste_lee_murphy @LindaEReimann @aurelio_fdez Personally, I am fine with terms like "statistically significant" IF we improve researchers statistical literacy AND educate them on approaches like equivalence testing that require them to also say what effects they would regard as practically relevant!

[Raphael Merz @RaphaelMerz]

@lukaswallrich @F_Bethke @ste_lee_murphy @LindaEReimann @aurelio_fdez I don't think confidence intervals alone would suffice. The statement still strongly implies that the groups *are the same*, even with CIs. To truly convince me, I'd need a prespecified smallest effect size of interest and a corresponding Equivalence Test.

[Raphael Merz @RaphaelMerz]

@F_Bethke @ste_lee_murphy @LindaEReimann @aurelio_fdez @trumanfrancis To strengthen the argument that there's no practically relevant effect of the intervention, you might consider Equivalence Testing with a predefined smallest effect size of interest (see @trumanfrancis response) (2/2)

[Raphael Merz @RaphaelMerz]

@F_Bethke @ste_lee_murphy @LindaEReimann @aurelio_fdez I would probably side with @trumanfrancis on this one and caution against concluding 'no substantial effect' based on nonsignificance –since statistical significance doesn't inform practical relevance. (1/2)

[Raphael Merz @RaphaelMerz]

@JakubTomek13 Nice! Excited to read about it! We were also left wondering how some of these really obvious misinterpretations made it through peer review and the editorial process… It’s surprising how often they "slip through"...

[Raphael Merz @RaphaelMerz]

@JakubTomek13 100% my experience! We actually considered including examples of these 'misinterpretation chains' - where one misinterpretation gets cited, then cited again, and so on. Didn’t make it into the final paper, but I think it’s definitely a real issue!