Sasha

[Sasha @unfriendlydata]

@HunterWieman Thanks! Five Markdown “extracts” that mimic the problem in the PDF you shared. Different topics, different formatting, slightly different wording. Not systematic, just to add some variation

Prompt is minimal: “evaluate the methodology.” Each model-condition is an independent run

[Sasha @unfriendlydata]

Got inspired by this and stress-tested whether LLMs catch this type of RDD estimand mismatch across five fake method sections

The GPT-5 family broadly is the clear winner, followed by Opus. Gemini mostly misses it

[Hunter Wieman @HunterWieman]

I have found a fatal flaw in a paper. It is an RDD paper, and the authors fundamentally misunderstand and misapply RDD. It's a very basic error, albeit a bit subtle.

Every AI model I have tried (including @RefineInk) cannot identify it, even when I tell them exactly where it is.

[Sasha @unfriendlydata]

Prompt was intentionally minimal: “Here is an extract from a PDF. Evaluate the methodology.”

[Sasha @unfriendlydata]

In the initial version models scored way better

I think that mostly comes from the vignettes being shorter, with fewer plausible issues to latch onto. Wording matters too, as well as whether the extract includes descriptive stats highlighting candidate differences

[Sasha @unfriendlydata]

@eigenrobot It endured and prevailed

[Sasha @unfriendlydata]

@shakoistsLog hypeskedasticity-robust standard errors

[Sasha @unfriendlydata]

@mikitaposts Вырашыў што галюцыную калі пабачыў б��сраўскі сцяг у стужцы

[Sasha @unfriendlydata]

Shoutout to a few of the 400+ people who made this possible: @ryancbriggs @mmmpsych @ecoInsubria @szaszibarnabas @BalazsAczel @mrtn_kvcs @GustavNilsonne @LiviaKosa @yaabdelfatah @oguzaliacar @ensaracem @BrianNosek

[Sasha @unfriendlydata]

Thanks to @OSFramework, everybody now has their Nature paper, and I want to share my interpretation of the results -- specifically, whether analytical robustness is higher or lower than expected and how to map the results into a broader garden-of-forking-paths discussion

[Dan Goldstein @dggoldst]

Nature meta-research project puts claims in social-science papers. I'm interested in Econ and Psych so I focused on that:

Econ had about the same rate of "not reproducible" analyses as Psych and a worse rate then Political Science.

https://t.co/UyIfeqt8CA

[Sasha @unfriendlydata]

Read the paper for details:

https://t.co/uZcp2jfsFE

[Sasha @unfriendlydata]

@dylanarmbruste3 @cremieuxrecueil "quasibasic statement" in a form "X affects Y" is trivially true in advance for almost any two variables in social/behavioral science and so carries no information, unlike probabilistic descriptions

Their framework makes more sense in a domain like physics with plausible nulls