Over half of psychology studies fail reproducibility test

[Vasilisa]

Over half of psychology studies fail reproducibility test
Largest replication study to date casts doubt on many published positive results.
Monya Baker
27 August 2015
Nature

Excerpt:
Don’t trust everything you read in the psychology literature. In fact, two thirds of it should probably be distrusted.

In the biggest project of its kind, Brian Nosek, a social psychologist and head of the Center for Open Science in Charlottesville, Virginia, and 269 co-authors repeated work reported in 98 original papers from three psychology journals, to see if they independently came up with the same results.

The studies they took on ranged from whether expressing insecurities perpetuates them to differences in how children and adults respond to fear stimuli, to effective ways to teach arithmetic.

According to the replicators' qualitative assessments, as previously reported by Nature, only 39 of the 100 replication attempts were successful. (There were 100 completed replication attempts on the 98 papers, as in two cases replication efforts were duplicated by separate teams.) But whether a replication attempt is considered successful is not straightforward. Today in Science, the team report the multiple different measures they used to answer this question1.

The 39% figure derives from the team's subjective assessments of success or failure (see graphic, 'Reliability test'). Another method assessed whether a statistically significant effect could be found, and produced an even bleaker result. Whereas 97% of the original studies found a significant effect, only 36% of replication studies found significant results. The team also found that the average size of the effects found in the replicated studies was only half that reported in the original studies.

There is no way of knowing whether any individual paper is true or false from this work, says Nosek. Either the original or the replication work could be flawed, or crucial differences between the two might be unappreciated. Overall, however, the project points to widespread publication of work that does not stand up to scrutiny.

Although Nosek is quick to say that most resources should be funnelled towards new research, he suggests that a mere 3% of scientific funding devoted to replication could make a big difference. The current amount, he says, is near-zero.

Replication failure
The work is part of the Reproducibility Project, launched in 2011 amid high-profile reports of fraud and faulty statistical analysis that led to an identity crisis in psychology.

John Ioannidis, an epidemiologist at Stanford University in California, says that the true replication-failure rate could exceed 80%, even higher than Nosek's study suggests. This is because the Reproducibility Project targeted work in highly respected journals, the original scientists worked closely with the replicators, and replicating teams generally opted for papers employing relatively easy methods — all things that should have made replication easier.

But, he adds, “We can really use it to improve the situation rather than just lament the situation. The mere fact that that collaboration happened at such a large scale suggests that scientists are willing to move in the direction of improving.”

The work published in Science is different from previous papers on replication because the team actually replicated such a large swathe of experiments, says Andrew Gelman, a statistician at Columbia University in New York. In the past, some researchers dismissed indications of widespread problems because they involved small replication efforts or were based on statistical simulations.

But they will have a harder time shrugging off the latest study, says Gelman. “This is empirical evidence, not a theoretical argument. The value of this project is that hopefully people will be less confident about their claims.”

Publication bias
The point, says Nosek, is not to critique individual papers but to gauge just how much bias drives publication in psychology.

< Full Story >

[SD45T-2]

Yet another reason why we are all doomed.

[Stek]

Still better than religion.

[Poki]

That's because alot try to follow black and white because of statistics. Averages, means, medians all belong in math. Psychology is the understanding of one of the most complicated subjects on earth as we are driven by what's around us as well as what's inside of us and they both have a cause and effect on each other which further compounds things. So while there may be a obsessive compulsive disorder it may have a different reason, desire, drive based on each person. To assume there is only a single path in and out is closed minded. Things can even compound on each other and look like something else. It is a very complicated subject that has to be taken on a situation by situation basis. Group concepts and theories are jumping points to dig deeper, not the answer or end all be all. Everything should be a "this is what's witnessed and observed" psychology shouldn't have conclusions, just shared knowledge of experience. It's to complicated to be simplified into a step a...b...c...d. it's more like a complicated Web with loops and back tracks and dead end paths.

[Lark]

Still better than religion.

How so?

[Poki]

How so?

Because religion just says "gay is bad" and uses a bible that has a conclusion that "gay is bad" as the reason. You can't even prove that the analysis is correct as there is no analysis, just conclusion. It is propogation of conclusions with no supporting evidence other then restatement of conclusion from another source. Atleast science, science of psychology included, has supporting evidence that denotes an attempt to reason and understand.

[Lark]

Because religion just says "gay is bad" and uses a bible that has a conclusion that "gay is bad" as the reason. You can't even prove that the analysis is correct as there is no analysis, just conclusion. It is propogation of conclusions with no supporting evidence other then restatement of conclusion from another source. Atleast science, science of psychology included, has supporting evidence that denotes an attempt to reason and understand.

That sounds unlike any religion I know.

It sounds a lot like what atheists say about religion though.

Should I accept the as an authority on something the people who declare a hatred for it? I'm not sure that is wise, I would not expect it to be balanced and unbiased.

[Poki]

That sounds unlike any religion I know.

It sounds a lot like what atheists say about religion though.

Should I accept the as an authority on something the people who declare a hatred for it? I'm not sure that is wise, I would not expect it to be balanced and unbiased.

Lol...you realize aethiest is based off of questioning due to lack of validity, proof, and reasoning. Compare this to all the blind "amen" followers that associate something good with God unquestionably because its what they want to believe, not because there is analysis, data, and proof to back it up. God is great, he allowed my car to drive with no engine....halleluja...amen...god is awesome. This man was stoned because he questioned god....halleluja...amen...he will smite those who go up against him. I am biased, biased against stupidity associated with blind faith. Which is exactly why this study was done against psychology. Can you imagine this being done against religion. It would not be embraced, all hell would break lose.

Accept what you want...this is nothing more then my reasonings based on my observations of so called "religious" people.

[Lark]

Lol...you realize aethiest is based off of questioning due to lack of validity, proof, and reasoning. Compare this to all the blind "amen" followers that associate something good with God unquestionably because its what they want to believe, not because there is analysis, data, and proof to back it up. God is great, he allowed my car to drive with no engine....halleluja...amen...god is awesome. This man was stoned because he questioned god....halleluja...amen...he will smite those who go up against him. I am biased, biased against stupidity associated with blind faith. Which is exactly why this study was done against psychology. Can you imagine this being done against religion. It would not be embraced, all hell would break lose.

Accept what you want...this is nothing more then my reasonings based on my observations of so called "religious" people.

You think that is a good basis from which to generalise?

I think you are talking about religion at its worst, are you prepared to consider religion at its best?

[Hard]

You think that is a good basis from which to generalise?

I think you are talking about religion at its worst, are you prepared to consider religion at its best?

The point made is very simple: Would you rather the study of the human mind be based off religon, or science?

The correct answer is science, even if there are flaws.