Why Most Published (medical) Research Findings Are False

[Octarine]

"Why Most Published Research Findings Are False"
http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124

Summary

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

See also:


"Why Current Publication Practices May Distort Science"
http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0050201

"Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias"
http://www.plosone.org/article/info:doi/10.1371/journal.pone.0003081

Is anyone concerned that our current medical practises and knowledge might be distorted due to these biases?

 

[CrystalViolet]

The thing is the more studies are done from different angles, the more likely the bias is going to correct itself. I think it's pretty poor practise to roll out new medication based on a few studies (which are more often than not done with commercial interests in mind.)
This is of course me shooting my mouth off before I've read the articles.
I will read and ponder futher, but I've been reading pretty extensively on the matter, as it seems to be of great concern in the scientific community at the moment.

 

[Octarine]

The thing is the more studies are done from different angles, the more likely the bias is going to correct itself.

Ideally yes. In practise, not necessarily because there are many biases as mentioned. Publication bias for example, means that negative findings (eg for antidepressants) are less likely to be published, resulting in a skewed publication record.

But on a deeper level, each scientific discipline tends to have established dogma - those who wish to do alternative studies find it more difficult to get funding and publish their studies.
Historians of science such as Kuhn (The Structure of Scientific Revolutions 1962) have pointed out a variety of biases and methodological suggestions (some of them are quite useful, such as building cumulative Bayesian models of the theory for example). But in general, scientists tend to ignore those outside their discipline.

 

[Betty Blue]

Ah yes, was thinking/talking about this recently (can't remember which) i have come to this conclusion myself.
I don't believe research to be false per say but rather the results are doctored to exaggerate findings... common practise i believe. Of course some will be completely falsified, but thats harder to get away with i imagine.
This is what i find humorous when arguing with people on fact versus anecdotal evidence.
Cite your stats and link me to research findings...lets see where they have gone wrong.

Of course it isn't always the case, just amusing to think of how rigidly people stick to stats... "It says so in the research stats...It is proof"

 

[Octarine]

"It says so in the research stats...It is proof"

Don't get me started on the cult of p=0.05.

 

[Jonny]

I don't know about you guys, but I find statistics to be one of the primary and most useful tools in research. Whether or not someone has a conflict of interest because of research funding, personal promotion, etc, and questionably "interprets" those statistics is another story. The P-value is an extremely useful part of statistical analysis.

 

[Octarine]

I don't know about you guys, but I find statistics to be one of the primary and most useful tools in research. Whether or not someone has a conflict of interest because of research funding, personal promotion, etc, and questionably "interprets" those statistics is another story. The P-value is an extremely useful part of statistical analysis.

p=0.05 is an arbitrary cutoff and often findings considered to be clinically useful because they met p<0.05, turn out to be false. There has even been a double blind RC homeopathy study that was published in the BMJ, which had 'statistically significant' findings for example.
The problem in general is the models on which the statistics are based are overly simplistic.

Secondly, the biases don't have to be explicit conflicts of interest. Biases exist even when the scientist acts ethically.
See also the 'decline effect'. It is not all bad though, p<0.05 in a small study is still useful because it is a useful indication that more research should be done.
But meta studies aren't necessarily much better, since they are based on studies with diverging methodology and the data can often only be treated in a superficial way.

But in general, the misuse/misunderstanding of statistics is why 'science based medicine' is starting to take ground from 'evidence based medicine'.

 

[swordpath]

What I want to know is if what today's guidelines say is considered a healthy blood pressure range is actually truly legitimate or not, because what's considered the healthy range has shifted pretty significantly over the past decade or two. I want to know if true medical research proves that what's considered healthy now is legit, or if it's a conspiracy to sell more drugs... Telling healthy people that their numbers "could be better", when in fact, they're perfectly fine.

 

[Octarine]

I'm not going to answer that question specifically, but it brings up a point. Clinical results such as blood tests are not reported correctly in clinical practice.
When a 'healthy' range is quoted, it should be based on a variety of factors, not limited to age and sex. The mean, SD of the norms should be quoted as well as mean error of the test itself.

 

[NickNaylor]

You can generally figure the motive when you look at who funded the study (ignore this post if its already been stated). I recently read an article that discussed how ADHD medications (dextroamphetamines, methylphenidate-stimulants) in youngsters actually aren't as bad on the heart as previously thought. This study was funded by pharmaceutical giant Shire, which coincidentially produces these medications.

Although you'd think that the ultimate motive IS public health, its unfortunate that the case is rather opposite. There is also a lot of speculative post-rationalization regarding health issues which seem to promote certain drugs to the public quite well. I am certain that a number of us can name more medications than we can flavours of chips.

 

[InvisibleJim]

The WHOs guidelines on meta-analysis is generally the best way to take many small results and to build something more conclusive. The biggest problem is that most 'high profile' studies are controversial for using loaded dice or are in very small sample sizes.

 

[onemoretime]

I knew social science wasn't in my future when no one could give me a good reason why the statistical calculations meant a damn thing, and were anything more than some numbers a person pulled out of his ass.

 

[Octarine]

The WHOs guidelines on meta-analysis is generally the best way to take many small results and to build something more conclusive. The biggest problem is that most 'high profile' studies are controversial for using loaded dice or are in very small sample sizes.

The problem is that even 'high profile' studies with large sample sizes can have flawed methodology, the results of which can be publicised as something it is not. It is possible for example to deviate substantially from a published protocol (relying instead on post-hoc thresholds) and still get published in a top journal such as NEJM or The Lancet.
But the flaws are not just limited to clinical trials.
One of the problems is that biomedical systems are complex and the methodology doesn't always help serve to eliminate other hypotheses.

 

[INTP]

this is ridiculous, i could expect this from someone who knows nothing about research, but from a doctor..

he is talking about bias being one big factor with this. eliminating bias by making the research setting is something that is #1 requirement for research setting..

naturally some drug company could want to create biased research settings, but the effects this sort of thing going public would be so huge to the company that it might even lose its licenses and it would definitely be prosecuted. researches getting to this sort of thing would never get a job again. all papers on the research are looked through by different authorities on medical field etc etc. so trying to do tricks like this isnt really a good idea, that is unless the government would want biased research on that particular topic also. like what happened with marijuana research back in the days, when they proved marijuana to be lethal and cause brain damage by suffocating some monkeys with marijuana smoke, open their brains and looking at their brains and seeing brain damage..

and in case you guys didnt know, other researchers can report biased studies to authorities and they will investigate it throughout. if the bias is intentional they will be prosecuted and if the researchers are such newbies that they didnt take care of the bias, they wont be able to get job very easy in the future, because they have kinda ruined their reputation on the field of research.

 

[Octarine]

As I keep saying much of the bias is not deliberate, at least on the individual level. You have to understand that in the real world, scientific research and especially medical research is both very messy and politicised. I could give you plenty of examples where governments ask for biased research to be done (or worse, research that is impossible), and this is not at all limited to drug trials.
Secondly, where there is a dominant scientific paradigm, it is often very easy to do the 'tricks' (eg not sticking to the protocol) that I mentioned above because it produces the result that everyone involved 'wanted' to see.

 

[Matthew_Z]

Type I errors are a bitch.

Remember kids, keep your α-levels under control.

 

[Octarine]

Sometimes it's not merely about alpha levels, but rather whether you are measuring the relevant confounding variables which may bias the result. When reading most papers, it is usually very easy to find potential biases that have not been fully accounted for.

Here is Dr Ioannidis' latest article:
"Comparison of Effect Sizes Associated With Biomarkers Reported in Highly Cited Individual Articles and in Subsequent Meta-analyses"
http://jama.ama-assn.org/content/305/21/2200.short

No significant biomarkers have been for cancer have made it to the clinic for the past 25 years, despite around $100 billion dollars spent on cancer research.