Tuesday, December 14, 2010

The truth wears off...

Jonah Lehrer has a fascinating article in the recent New Yorker which describes in detail a disturbing trend:
..all sorts of well-established multiply confirmed findings have started to look increasingly uncertain, It's as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn't yet have an official name, but it's occurring across a wide range of fields, from psychology to ecology. In the field of medicine the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies from cardiac stents to Vitamin E and antidepressants...a forthcoming analysis demonstrates that the efficacy of antidepressants has gone down as much as three-fold in recent decades.
Lehrer tells the story of a number of serious scientists who have reported statistically significant effects with appropriate controls, only to find then disappear over time, seemingly iron-clad results that on repetition seemed to fade away. One example is "verbal overshadowing", subjects being shown a face and asked to describe it being much less likely to recognize the face when shown it later than those who had simply looked at it. Another theory that has fallen apart is the claim that females use symmetry as a proxy for reproductive fitness of males. A 2005 study found that of the 50 most cited clinical research studies (with randomized control trials), almost half were subsequently not replicated or had their effects significantly downgraded, and these studies had guided clinical practice (hormone replacement therapy for menopausal women, low-dose aspirin to prevent heart attacks and strokes).

It is not entirely clear why this is happening, and several possibilities are mentioned:
-statistical regression to the mean, an early statistical fluke gets canceled out.
-publication bias on the part of journals, who prefer positive data over null results
-selective reporting, or significance chasing.  A review found that over 90% of psychological studies reporting statistically significant data, i.e. odds of being produced by chance less than 5% of the time, found the effect they were looking for. (One classic example of selective reporting concerns testing acupuncture in Asian countries - largely positive data - versus Western countries - less than half confirming. See today's other posting on MindBlog).

The problem of selective reporting doesn't derive necessarily from dishonesty, but from the fundamental cognitive flaw that we like proving ourselves right and hate being wrong.  The decline effect may actually be a decline of illusion.

We shouldn't throw out the baby with the bath water, as Lehrer notes in a subsequent blog posting. These problems don't mean we shouldn't believe in evolution or climate change:
One of the sad ironies of scientific denialism is that we tend to be skeptical of precisely the wrong kind of scientific claims. In poll after poll, Americans have dismissed two of the most robust and widely tested theories of modern science: evolution by natural selection and climate change. These are theories that have been verified in thousands of different ways by thousands of different scientists working in many different fields. (This doesn’t mean, of course, that such theories won’t change or get modified – the strength of science is that nothing is settled.) Instead of wasting public debate on creationism or the rhetoric of Senator Inhofe, I wish we’d spend more time considering the value of spinal fusion surgery, or second generation antipsychotics, or the verity of the latest gene association study.

5 comments:

  1. Isn't it also possible that the reality just changes over time?

    ReplyDelete
  2. Anonymous2:11 PM

    Excellent answer to The Decline Effect

    http://www.sciencebasedmedicine.org/?p=8987

    ReplyDelete
  3. Anonymous12:18 PM

    It could also be that postmodernist ideology has diffused to people who should never be given a set of strong ideas in the first place........

    ReplyDelete
  4. Anonymous5:05 AM

    It could be a little something to do with quantum entanglement.
    The universe is a much odder place than the scientific materialists would like to think.

    ReplyDelete
  5. Anonymous9:27 AM

    A lie is believed if it something that one wants to believe even before it is presented. Any person or obect will easily fall in the direction it is leaning but it requires a far greater force to make it fall in the opposite direction. That which begins out of nowhere and ends up nowhere is not reality.

    ReplyDelete