“Torture data long enough and it will confess to anything.” Then, once you get the confession you want, you can HARK, or hypothesize after the results are known — Ah, yes, this is what we expected to find all along. I knew the pill would help fight high blood pressure. In a situation like this, though, there’s a serious risk that what you’re looking at is not a pill that lowers high blood pressure, but statistical noise.

The good news is that there’s a growing awareness among researchers in psychology and other fields affected by replication crises of how these practices can generate weak and non-replicable results. Science reformers have begun constructing guardrails that effectively constrict researchers and reduce their degrees of freedom: For instance, you can incentivize or require researchers to “preregister” their hypotheses and lay out exactly which statistical tests they plan to run, so that if they change their analysis plan or hypothesis midstream, this will be visible for all the world to see. You can also incentivize or require them to share their data, which makes it easier for other researchers to check and see whether they engaged in statistical chicanery.

I think there’s a reasonably strong case to be made that the positive results reported in “Psychosocial Functioning in Transgender Youth after 2 Years of Hormones,” a highly anticipated research paper just published in The New England Journal of Medicine, can be at least partially explained by the sort of statistical cherry-picking that tends to generate wobbly findings.

Singal is brilliant & meticulous. Read the whole post at https://jessesingal.substack.com/p/on-scientific-transparency-researcher