Daniele Fanelli

Meta-metascientist, methodical methodologist, integral integrity expert, conscientious consciousness student, ass Prof @ Heriot-Watt, everyone @comCensus.org

[Daniele Fanelli @DFanDaBiasedMan]

22 minutes in, and this seems like a pretty good vision to me. https://t.co/ZwDUjso66v

[Daniele Fanelli @DFanDaBiasedMan]

Those who have been peddling the idea of a replication crisis will be happy, I hope. But if critics like myself are right, and the crisis is misconstrued and overblown, then this will waste many resources that could have been invested on new discoveries.

[Stuart Buck @stuartbuck1]

Interesting piece on replication in the news (it quotes me): https://t.co/m5gtKGaYxo

[Daniele Fanelli @DFanDaBiasedMan]

Last day to apply for our postdoc on complexity and integrity. LLM, NLP enthusiasts are especially encouraged to apply. PhD-equivalent experience is acceptable, https://t.co/fuNwyuf9YY

[Daniele Fanelli @DFanDaBiasedMan]

Last 38 hours to apply for a short term postdoc on research integrity and complexity. LLM and NLP enthusiast especially welcome! https://t.co/fuNwyuf9YY

[Daniele Fanelli @DFanDaBiasedMan]

Want to apply your NLP and LLM skills to Metascientific questions? Join us for 6 months, during Edinburgh's festival season! https://t.co/fuNwyuf9YY

[Daniele Fanelli @DFanDaBiasedMan]

Want to apply your NLP and LLM skills to Metascientific questions? Join us for 6 months, during Edinburgh's festival season! https://t.co/fuNwyuf9YY

[Daniele Fanelli @DFanDaBiasedMan]

Last day to apply for a Metascience postdoc in our lab, short-term but very flexible about topic! https://t.co/Hr1kNGeTlj

[Daniele Fanelli @DFanDaBiasedMan]

Deadline TOMORROW night for a METASCIENCE postdoc, short term but fairly open in scope: https://t.co/Hr1kNGelvL

[Daniele Fanelli @DFanDaBiasedMan]

SHORT-TERM POSTDOC at TEMET lab: We are looking for an enthusiastic metascientist to help us develop methods to measure the complexity of studies. Less than a year left but Summer is the best (festival) period in Edinburgh! Please circulate. https://t.co/Hr1kNGelvL

[Daniele Fanelli @DFanDaBiasedMan]

Lee, sorry but I'll move on to more productive activities. I am sure you have better things to attend, too. All you needed to do to show no dogmatism was to acknowledge that, once I showed to you that Protzko et al (which you set as your best evidence) does not show high rep as you thought, then your belief in a rep problem was logically weakened. Instead, you seem to keep changing topic, eluding the point. My instinct is to be disappointed, but I am happy to call it a misunderstanding. Clearly X is not a good medium for intelligent scientific conversations. Cheerio.

[Daniele Fanelli @DFanDaBiasedMan]

@PsychRabble @BrettButtliere Lee, I hold your work in great esteem. But it seems clear to me at this point that you'll do anything but answer my very simple and direct question. It makes me think that
@BrettButtliere
is right, unfortunately.

[Daniele Fanelli @DFanDaBiasedMan]

@PsychRabble what evdence would convince you that the replication crisis is exaggerated? Your argument, as I understand it so far: anything below 70% (why 70??) reproducibility is too low, and yes there is a crisis. My argument: it really depends on the field; immature (low-codified) fields dealing with highly complex (variable in space and time) phenomena may have low reproducibility for perfectly fair reasons. Unless we know how much we should expect for a specific literature, claims of "crisis" and QRPs are unwarranted and unjustified. I would change my mind if I saw systematically low reproducibility in most literatures, regardless of phenomena and maturity. To the contrary, the numbers produced by replication studies show >>50%, and often >>80% in virtually all cases. The lowest figures are for social psychology, unsurprisingly (high complexity). This evidence, a long time ago, made me change my mind and figure that this crisis narrative is overblown. I am sure plenty of literature in psychology and elsewhere is rubbish, but even in social psych average repr. is much higher than you could pessimistically expect given the complexity involved. Over to you: what evidence makes you think that rep is dismally low (everywhere in science??) and what evidence would make you change your mind?

[Daniele Fanelli @DFanDaBiasedMan]

@PsychRabble It is not dogmatism if people are open to changing their minds faced with new evidence. Now, is the new evidence that you just learned about Protzko et al not showing high rep. changing your mind at least a bit?

[Daniele Fanelli @DFanDaBiasedMan]

@PsychRabble Ok, thanks. So, you say, if they did not succeed in showing 90%, this would change your mind. right? So, now that you know that they did NOT achieve that at all - and it is not my opinion but documented fact - shouldn't you start to change your mind?

[Daniele Fanelli @DFanDaBiasedMan]

@PsychRabble Your response was "I've answered the "what could change my mind?" q several times now. https://t.co/mHrVmi4YsB…" . And that post cites Protzko. Would you be so kind to articulate?

[The Dark Fiddling Pirate Jussim @PsychRabble]

Let's get specific. Right about what? I've already given what it would take for me to conclude "this set of studies shows high replicability." And given one example: The Nature paper (and afaict, its Schroedinger's preregistration clusterf*ck does not change its high replic rate). Again, if you want to dismiss the Nature paper, that's fine, but its irrelevant to your q here, which is: "what evidence/analysis would conivnce you that you are wrong...?" Now, repeat that over and over and over and over, and I 'd say a field is fine.

You say, "Well I just gave you that evidence."
I say: "Well no. You just gave me evidence that you say meets your standards. When I checked it, it definitely did not meet my standards. You and I using different standards does not mean either one of us is particularly dogmatic."

[Daniele Fanelli @DFanDaBiasedMan]

Ok, then back to Protzko et al. The real replication rate they observed, with ordinary analyses (and not the fanciful ones that were claimed to be pre-registered and weren't) was not 90%, as you think, but at best 70% (see https://t.co/inZf1vdaM9 and https://t.co/eDIn9T9F8D). 70% was attained by plenty of replication studies before (see my thread, and Coleman's blog). Moreover, the Protzko study was not designed to prove causality. So - apart from its retraction - the study proves the contrary of what you are claiming . Because the rates they observed are in the same ballpark than those already observed in the literature by many replication studies. Do you see? And 70% is also your threshold for good science, right? Which was attained by plenty of replication studies already. Shouldn't all this count against your hypothesis?

[Daniele Fanelli @DFanDaBiasedMan]

@PsychRabble @lakens LOL - you may be right, but surely you can see how I get the same impression from you. I'll ask here again the key question: what evidence/analysis would convince you that you are wrong and I am right?

[Daniele Fanelli @DFanDaBiasedMan]

@PsychRabble @lakens And there are more studies now, that show even higher rates. Let me ask you this: what evidence would convince you that you are wrong?

[Daniele Fanelli @DFanDaBiasedMan]

@PsychRabble @lakens The point being that all of those studies are at least compatible with 70% or higher. and most suggest >50% minimum. These studies should be known to all - I didn't make them up.