Those who have been peddling the idea of a replication crisis will be happy, I hope. But if critics like myself are right, and the crisis is misconstrued and overblown, then this will waste many resources that could have been invested on new discoveries.
Last day to apply for our postdoc on complexity and integrity. LLM, NLP enthusiasts are especially encouraged to apply. PhD-equivalent experience is acceptable, https://t.co/fuNwyuf9YY
Last 38 hours to apply for a short term postdoc on research integrity and complexity. LLM and NLP enthusiast especially welcome! https://t.co/fuNwyuf9YY
SHORT-TERM POSTDOC at TEMET lab: We are looking for an enthusiastic metascientist to help us develop methods to measure the complexity of studies. Less than a year left but Summer is the best (festival) period in Edinburgh! Please circulate. https://t.co/Hr1kNGelvL
Lee, sorry but I'll move on to more productive activities. I am sure you have better things to attend, too. All you needed to do to show no dogmatism was to acknowledge that, once I showed to you that Protzko et al (which you set as your best evidence) does not show high rep as you thought, then your belief in a rep problem was logically weakened. Instead, you seem to keep changing topic, eluding the point. My instinct is to be disappointed, but I am happy to call it a misunderstanding. Clearly X is not a good medium for intelligent scientific conversations. Cheerio.
@PsychRabble@BrettButtliere Lee, I hold your work in great esteem. But it seems clear to me at this point that you'll do anything but answer my very simple and direct question. It makes me think that
@BrettButtliere
is right, unfortunately.
@PsychRabble what evdence would convince you that the replication crisis is exaggerated? Your argument, as I understand it so far: anything below 70% (why 70??) reproducibility is too low, and yes there is a crisis. My argument: it really depends on the field; immature (low-codified) fields dealing with highly complex (variable in space and time) phenomena may have low reproducibility for perfectly fair reasons. Unless we know how much we should expect for a specific literature, claims of "crisis" and QRPs are unwarranted and unjustified. I would change my mind if I saw systematically low reproducibility in most literatures, regardless of phenomena and maturity. To the contrary, the numbers produced by replication studies show >>50%, and often >>80% in virtually all cases. The lowest figures are for social psychology, unsurprisingly (high complexity). This evidence, a long time ago, made me change my mind and figure that this crisis narrative is overblown. I am sure plenty of literature in psychology and elsewhere is rubbish, but even in social psych average repr. is much higher than you could pessimistically expect given the complexity involved. Over to you: what evidence makes you think that rep is dismally low (everywhere in science??) and what evidence would make you change your mind?
@PsychRabble It is not dogmatism if people are open to changing their minds faced with new evidence. Now, is the new evidence that you just learned about Protzko et al not showing high rep. changing your mind at least a bit?
@PsychRabble Ok, thanks. So, you say, if they did not succeed in showing 90%, this would change your mind. right? So, now that you know that they did NOT achieve that at all - and it is not my opinion but documented fact - shouldn't you start to change your mind?
@PsychRabble Your response was "I've answered the "what could change my mind?" q several times now. https://t.co/mHrVmi4YsB…" . And that post cites Protzko. Would you be so kind to articulate?
Let's get specific. Right about what? I've already given what it would take for me to conclude "this set of studies shows high replicability." And given one example: The Nature paper (and afaict, its Schroedinger's preregistration clusterf*ck does not change its high replic rate). Again, if you want to dismiss the Nature paper, that's fine, but its irrelevant to your q here, which is: "what evidence/analysis would conivnce you that you are wrong...?" Now, repeat that over and over and over and over, and I 'd say a field is fine.
You say, "Well I just gave you that evidence."
I say: "Well no. You just gave me evidence that you say meets your standards. When I checked it, it definitely did not meet my standards. You and I using different standards does not mean either one of us is particularly dogmatic."
Ok, then back to Protzko et al. The real replication rate they observed, with ordinary analyses (and not the fanciful ones that were claimed to be pre-registered and weren't) was not 90%, as you think, but at best 70% (see https://t.co/inZf1vdaM9 and https://t.co/eDIn9T9F8D). 70% was attained by plenty of replication studies before (see my thread, and Coleman's blog). Moreover, the Protzko study was not designed to prove causality. So - apart from its retraction - the study proves the contrary of what you are claiming . Because the rates they observed are in the same ballpark than those already observed in the literature by many replication studies. Do you see? And 70% is also your threshold for good science, right? Which was attained by plenty of replication studies already. Shouldn't all this count against your hypothesis?
@PsychRabble@lakens LOL - you may be right, but surely you can see how I get the same impression from you. I'll ask here again the key question: what evidence/analysis would convince you that you are wrong and I am right?
@PsychRabble@lakens And there are more studies now, that show even higher rates. Let me ask you this: what evidence would convince you that you are wrong?
@PsychRabble@lakens The point being that all of those studies are at least compatible with 70% or higher. and most suggest >50% minimum. These studies should be known to all - I didn't make them up.