Thursday, January 30, 2014

Most published scientific results are false.

I would highly recommend reading this article by George Johnson, which points in particular to the work of John P. A. Ioannidis, a kind of meta-scientist who researches research, who wrote a 2005 paper pointedly titled “Why Most Published Research Findings Are False.” Here is one clip from the Johnson article:
If one of five competing labs is alone in finding an effect, that result is the one likely to be published. But there is a four in five chance that it is wrong. Papers reporting negative conclusions are more easily ignored...Putting all of this together, Dr. Ioannidis devised a mathematical model supporting the conclusion that most published findings are probably incorrect....the same year he published another blockbuster, examining more than a decade’s worth of highly regarded papers — the effect of a daily aspirin on cardiac disease, for example, or the risks of hormone replacement therapy for older women. He found that a large proportion of the conclusions were undermined or contradicted by later studies.
His work was just the beginning. Concern about the problem has reached the point that the journal Nature has assembled an archive, filled with reports and analyses, called Challenges in Irreproducible Research.. Among them is a paper in which C. Glenn Begley, who is chief scientific officer at TetraLogic Pharmaceuticals, described an experience he had while at Amgen, another drug company. He and his colleagues could not replicate 47 of 53 landmark papers about cancer. Some of the results could not be reproduced even with the help of the original scientists working in their own labs....Given what is at stake, it seems like a moral failing that the titles of the papers were not revealed. That was forbidden, we’re told, by confidentiality agreements imposed by the labs.