Over half of psychology studies fail reproducibility test
Largest replication study to date casts doubt on many published positive results.
Monya Baker
27 August 2015
Nature
Excerpt:
Don’t trust everything you read in the psychology literature. In fact, two thirds of it should probably be distrusted.
In the biggest project of its kind, Brian Nosek, a social psychologist and head of the Center for Open Science in Charlottesville, Virginia, and 269 co-authors repeated work reported in 98 original papers from three psychology journals, to see if they independently came up with the same results.
The studies they took on ranged from whether expressing insecurities perpetuates them to differences in how children and adults respond to fear stimuli, to effective ways to teach arithmetic.
According to the replicators' qualitative assessments, as previously reported by Nature, only 39 of the 100 replication attempts were successful. (There were 100 completed replication attempts on the 98 papers, as in two cases replication efforts were duplicated by separate teams.) But whether a replication attempt is considered successful is not straightforward. Today in Science, the team report the multiple different measures they used to answer this question1.
The 39% figure derives from the team's subjective assessments of success or failure (see graphic, 'Reliability test'). Another method assessed whether a statistically significant effect could be found, and produced an even bleaker result. Whereas 97% of the original studies found a significant effect, only 36% of replication studies found significant results. The team also found that the average size of the effects found in the replicated studies was only half that reported in the original studies.
There is no way of knowing whether any individual paper is true or false from this work, says Nosek. Either the original or the replication work could be flawed, or crucial differences between the two might be unappreciated. Overall, however, the project points to widespread publication of work that does not stand up to scrutiny.
Although Nosek is quick to say that most resources should be funnelled towards new research, he suggests that a mere 3% of scientific funding devoted to replication could make a big difference. The current amount, he says, is near-zero.
Replication failure
The work is part of the Reproducibility Project, launched in 2011 amid high-profile reports of fraud and faulty statistical analysis that led to an identity crisis in psychology.
John Ioannidis, an epidemiologist at Stanford University in California, says that the true replication-failure rate could exceed 80%, even higher than Nosek's study suggests. This is because the Reproducibility Project targeted work in highly respected journals, the original scientists worked closely with the replicators, and replicating teams generally opted for papers employing relatively easy methods — all things that should have made replication easier.
But, he adds, “We can really use it to improve the situation rather than just lament the situation. The mere fact that that collaboration happened at such a large scale suggests that scientists are willing to move in the direction of improving.”
The work published in Science is different from previous papers on replication because the team actually replicated such a large swathe of experiments, says Andrew Gelman, a statistician at Columbia University in New York. In the past, some researchers dismissed indications of widespread problems because they involved small replication efforts or were based on statistical simulations.
But they will have a harder time shrugging off the latest study, says Gelman. “This is empirical evidence, not a theoretical argument. The value of this project is that hopefully people will be less confident about their claims.”
Publication bias
The point, says Nosek, is not to critique individual papers but to gauge just how much bias drives publication in psychology.
< Full Story >
User Tag List
-
08-29-2015, 10:36 PM #1
Over half of psychology studies fail reproducibility test
the formless thing which gives things form!
Found Forum Haiku Project
Positive Spin | your feedback welcomed | Darker Criticism
-
08-29-2015, 11:41 PM #2
Yet another reason why we are all doomed.
1w2-6w5-3w2 so/sp
"I took one those personality tests. It came back negative." - Dan Mintz
-
09-03-2015, 01:45 AM #3
-
09-03-2015, 08:56 AM #4
- Join Date
- Dec 2008
- MBTI
- STP
- Posts
- 10,583
That's because alot try to follow black and white because of statistics. Averages, means, medians all belong in math. Psychology is the understanding of one of the most complicated subjects on earth as we are driven by what's around us as well as what's inside of us and they both have a cause and effect on each other which further compounds things. So while there may be a obsessive compulsive disorder it may have a different reason, desire, drive based on each person. To assume there is only a single path in and out is closed minded. Things can even compound on each other and look like something else. It is a very complicated subject that has to be taken on a situation by situation basis. Group concepts and theories are jumping points to dig deeper, not the answer or end all be all. Everything should be a "this is what's witnessed and observed" psychology shouldn't have conclusions, just shared knowledge of experience. It's to complicated to be simplified into a step a...b...c...d. it's more like a complicated Web with loops and back tracks and dead end paths.
Im out, its been funLark liked this post
-
09-03-2015, 09:06 AM #5
-
09-03-2015, 09:17 AM #6
- Join Date
- Dec 2008
- MBTI
- STP
- Posts
- 10,583
Because religion just says "gay is bad" and uses a bible that has a conclusion that "gay is bad" as the reason. You can't even prove that the analysis is correct as there is no analysis, just conclusion. It is propogation of conclusions with no supporting evidence other then restatement of conclusion from another source. Atleast science, science of psychology included, has supporting evidence that denotes an attempt to reason and understand.
Im out, its been fun
-
09-03-2015, 10:45 AM #7
- Join Date
- Jun 2009
- Posts
- 27,607
-
09-03-2015, 11:11 AM #8
- Join Date
- Dec 2008
- MBTI
- STP
- Posts
- 10,583
Lol...you realize aethiest is based off of questioning due to lack of validity, proof, and reasoning. Compare this to all the blind "amen" followers that associate something good with God unquestionably because its what they want to believe, not because there is analysis, data, and proof to back it up. God is great, he allowed my car to drive with no engine....halleluja...amen...god is awesome. This man was stoned because he questioned god....halleluja...amen...he will smite those who go up against him. I am biased, biased against stupidity associated with blind faith. Which is exactly why this study was done against psychology. Can you imagine this being done against religion. It would not be embraced, all hell would break lose.
Accept what you want...this is nothing more then my reasonings based on my observations of so called "religious" people.Im out, its been funHard liked this post
-
09-03-2015, 11:36 AM #9
- Join Date
- Jun 2009
- Posts
- 27,607
-
09-03-2015, 11:43 AM #10
- Join Date
- Jan 2014
- MBTI
- ENFJ
- Enneagram
- 1w2 sp/so
- Socionics
- EIE Fe
- Posts
- 8,589
Julius_Van_Der_Beak liked this post
Similar Threads
-
The value of random on-line personality tests
By Ilah in forum General PsychologyReplies: 6Last Post: 09-26-2008, 08:33 PM