When Being Right Less Than Half the Time Is … Fine

Let’s say you do a job that involves making predictions about human behavior — you manage money, you sell things, you write opinion columns. Just less than half of your predictions turn out to be more or less right, about 10% are completely wrong, and with the rest it’s hard to say for sure. Would a success rate like that make you good at your job?

This ran through my mind as I perused the findings of the Center for Open Science’s huge Systematizing Confidence in Open Research and Evidence project, which were published in a series of articles in the journal Nature at the beginning of this month and are available outside the paywall — along with other papers and supporting data — at the center’s website. In a study that attempted to replicate the findings of 164 randomly selected articles published in social science journals using new data sets, 49.3% of the replications “had statistically significant findings with the same pattern as the original finding,” 9.7% showed an “opposing pattern” and 40.4% showed no statistically significant effect.