There's been a lot of discussion lately regarding the "replication crisis" in science at the moment, where researchers have been testing Ioannidis' argument that most published research findings are false. The basic idea here being that there are a number of features inherent to scientific processes, as well as a collection of bad scientific practices, that culminate in a number of reported positive findings sneaking into the literature and not being corrected until it's already been cited as evidence of a false claim throughout entire fields of science.
For example, one major issue is the problem of publication bias. This describes the effect of journals favouring positive or "sexy" results over studies that get negative results or report finding no effect where previous results had reported effects. So what ends up happening is that ten researchers might choose to research whether X causes Y, and eight of them find no effect but two find an effect. This can happen due to bad methodology, fraud, experimenter bias, or just simply chance and statistical noise. But as a result of studies favouring positive findings, those two papers will get published and the eight negative findings will get rejected. Since they're never published, a review of the published literature will tell us that 100% of studies on the topic find a result, and without getting access to the unpublished studies we'd be unaware that only 20% found a result - which is much less impressive.
THE QRPs ARE COMING FROM INSIDE THE HOUSE
So far we can see that there is a serious issue here that needs addressing but it doesn't seem like all hope is lost. Things like publication bias, once identified, seem fairly easy to address - encourage journals to publish negative findings, start journals for negative findings, emphasise the importance of negative findings when teaching students to avoid the assumption that they're a waste of time, and so on. Most of these problems have relatively easy solutions because everyone agrees that it's bad science and weakens the confidence in our published results.
But what do we do when the people doing bad science think that they're doing good science?
