"Lies, Damned Lies, and Medical Science": Why you may want a healthy level of skepticism when looking at research

Here's another article to raise your caution quotient about the science research you read. I hope you click through and read the whole article rather than just my excerpt. From "Lies, Damned Lies, and Medical Science" (The Atlantic):

When a five-year study of 10,000 people finds that those who take more vitamin X are less likely to get cancer Y, you’d think you have pretty good reason to take more vitamin X, and physicians routinely pass these recommendations on to patients. But these studies often sharply conflict with one another. Studies have gone back and forth on the cancer-preventing powers of vitamins A, D, and E; on the heart-health benefits of eating fat and carbs; and even on the question of whether being overweight is more likely to extend or shorten your life. How should we choose among these dueling, high-profile nutritional findings? [John] Ioannidis suggests a simple approach: ignore them all.

...

THOUGH SCIENTISTS AND science journalists are constantly talking up the value of the peer-review process, researchers

 

admit among themselves that biased, erroneous, and even blatantly fraudulent studies easily slip through it. Nature, the grande dame of science journals, stated in a 2006 editorial, “Scientists understand that peer review per se provides only a minimal assurance of quality, and that the public conception of peer review as a stamp of authentication is far from the truth.” What’s more, the peer-review process often pressures researchers to shy away from striking out in genuinely new directions, and instead to build on the findings of their colleagues (that is, their potential reviewers) in ways that only seem like breakthroughs—as with the exciting-sounding gene linkages (autism genes identified!) and nutritional findings (olive oil lowers blood pressure!) that are really just dubious and conflicting variations on a theme.

Most journal editors don’t even claim to protect against the problems that plague these studies. University and government research overseers rarely step in to directly enforce research quality, and when they do, the science community goes ballistic over the outside interference. The ultimate protection against research error and bias is supposed to come from the way scientists constantly retest each other’s results—except they don’t. Only the most prominent findings are likely to be put to the test, because there’s likely to be publication payoff in firming up the proof, or contradicting it.

But even for medicine’s most influential studies, the evidence sometimes remains surprisingly narrow. ...

Click to read the whole article.

Comments

[Steve]

Not to mention the whole idea of effect size, absolute risk, etc...

[Daniel]

In other words, ignore everything and take your chances?