• You are currently viewing our forum as a guest, which gives you limited access to view most discussions and access our other features. By joining our free community, you will have access to additional post topics, communicate privately with other members (PM), view blogs, respond to polls, upload content, and access many other special features. Registration is fast, simple and absolutely free, so please join our community today! Just click here to register. You should turn your Ad Blocker off for this site or certain features may not work properly. If you have any problems with the registration process or your account login, please contact us by clicking here.

Over half of psychology studies fail reproducibility test

Vasilisa

Symbolic Herald
Joined
Feb 2, 2010
Messages
3,946
Instinctual Variant
so/sx
Over half of psychology studies fail reproducibility test
Largest replication study to date casts doubt on many published positive results.
Monya Baker
27 August 2015
Nature

Excerpt:
Don’t trust everything you read in the psychology literature. In fact, two thirds of it should probably be distrusted.

In the biggest project of its kind, Brian Nosek, a social psychologist and head of the Center for Open Science in Charlottesville, Virginia, and 269 co-authors repeated work reported in 98 original papers from three psychology journals, to see if they independently came up with the same results.

The studies they took on ranged from whether expressing insecurities perpetuates them to differences in how children and adults respond to fear stimuli, to effective ways to teach arithmetic.

According to the replicators' qualitative assessments, as previously reported by Nature, only 39 of the 100 replication attempts were successful. (There were 100 completed replication attempts on the 98 papers, as in two cases replication efforts were duplicated by separate teams.) But whether a replication attempt is considered successful is not straightforward. Today in Science, the team report the multiple different measures they used to answer this question1.

The 39% figure derives from the team's subjective assessments of success or failure (see graphic, 'Reliability test'). Another method assessed whether a statistically significant effect could be found, and produced an even bleaker result. Whereas 97% of the original studies found a significant effect, only 36% of replication studies found significant results. The team also found that the average size of the effects found in the replicated studies was only half that reported in the original studies.

There is no way of knowing whether any individual paper is true or false from this work, says Nosek. Either the original or the replication work could be flawed, or crucial differences between the two might be unappreciated. Overall, however, the project points to widespread publication of work that does not stand up to scrutiny.

Although Nosek is quick to say that most resources should be funnelled towards new research, he suggests that a mere 3% of scientific funding devoted to replication could make a big difference. The current amount, he says, is near-zero.

Replication failure
The work is part of the Reproducibility Project, launched in 2011 amid high-profile reports of fraud and faulty statistical analysis that led to an identity crisis in psychology.

John Ioannidis, an epidemiologist at Stanford University in California, says that the true replication-failure rate could exceed 80%, even higher than Nosek's study suggests. This is because the Reproducibility Project targeted work in highly respected journals, the original scientists worked closely with the replicators, and replicating teams generally opted for papers employing relatively easy methods — all things that should have made replication easier.

But, he adds, “We can really use it to improve the situation rather than just lament the situation. The mere fact that that collaboration happened at such a large scale suggests that scientists are willing to move in the direction of improving.”

The work published in Science is different from previous papers on replication because the team actually replicated such a large swathe of experiments, says Andrew Gelman, a statistician at Columbia University in New York. In the past, some researchers dismissed indications of widespread problems because they involved small replication efforts or were based on statistical simulations.

But they will have a harder time shrugging off the latest study, says Gelman. “This is empirical evidence, not a theoretical argument. The value of this project is that hopefully people will be less confident about their claims.”

Publication bias
The point, says Nosek, is not to critique individual papers but to gauge just how much bias drives publication in psychology.

< Full Story >
 

SD45T-2

Senior Jr.
Joined
Feb 18, 2012
Messages
4,229
MBTI Type
ESTJ
Enneagram
1w2
Instinctual Variant
so/sp
Yet another reason why we are all doomed. :laugh:
 

Poki

New member
Joined
Dec 4, 2008
Messages
10,436
MBTI Type
STP
Instinctual Variant
sx/so
That's because alot try to follow black and white because of statistics. Averages, means, medians all belong in math. Psychology is the understanding of one of the most complicated subjects on earth as we are driven by what's around us as well as what's inside of us and they both have a cause and effect on each other which further compounds things. So while there may be a obsessive compulsive disorder it may have a different reason, desire, drive based on each person. To assume there is only a single path in and out is closed minded. Things can even compound on each other and look like something else. It is a very complicated subject that has to be taken on a situation by situation basis. Group concepts and theories are jumping points to dig deeper, not the answer or end all be all. Everything should be a "this is what's witnessed and observed" psychology shouldn't have conclusions, just shared knowledge of experience. It's to complicated to be simplified into a step a...b...c...d. it's more like a complicated Web with loops and back tracks and dead end paths.
 

Poki

New member
Joined
Dec 4, 2008
Messages
10,436
MBTI Type
STP
Instinctual Variant
sx/so

Because religion just says "gay is bad" and uses a bible that has a conclusion that "gay is bad" as the reason. You can't even prove that the analysis is correct as there is no analysis, just conclusion. It is propogation of conclusions with no supporting evidence other then restatement of conclusion from another source. Atleast science, science of psychology included, has supporting evidence that denotes an attempt to reason and understand.
 

Lark

Active member
Joined
Jun 21, 2009
Messages
29,569
Because religion just says "gay is bad" and uses a bible that has a conclusion that "gay is bad" as the reason. You can't even prove that the analysis is correct as there is no analysis, just conclusion. It is propogation of conclusions with no supporting evidence other then restatement of conclusion from another source. Atleast science, science of psychology included, has supporting evidence that denotes an attempt to reason and understand.

That sounds unlike any religion I know.

It sounds a lot like what atheists say about religion though.

Should I accept the as an authority on something the people who declare a hatred for it? I'm not sure that is wise, I would not expect it to be balanced and unbiased.
 

Poki

New member
Joined
Dec 4, 2008
Messages
10,436
MBTI Type
STP
Instinctual Variant
sx/so
That sounds unlike any religion I know.

It sounds a lot like what atheists say about religion though.

Should I accept the as an authority on something the people who declare a hatred for it? I'm not sure that is wise, I would not expect it to be balanced and unbiased.

Lol...you realize aethiest is based off of questioning due to lack of validity, proof, and reasoning. Compare this to all the blind "amen" followers that associate something good with God unquestionably because its what they want to believe, not because there is analysis, data, and proof to back it up. God is great, he allowed my car to drive with no engine....halleluja...amen...god is awesome. This man was stoned because he questioned god....halleluja...amen...he will smite those who go up against him. I am biased, biased against stupidity associated with blind faith. Which is exactly why this study was done against psychology. Can you imagine this being done against religion. It would not be embraced, all hell would break lose.

Accept what you want...this is nothing more then my reasonings based on my observations of so called "religious" people.
 

Lark

Active member
Joined
Jun 21, 2009
Messages
29,569
Lol...you realize aethiest is based off of questioning due to lack of validity, proof, and reasoning. Compare this to all the blind "amen" followers that associate something good with God unquestionably because its what they want to believe, not because there is analysis, data, and proof to back it up. God is great, he allowed my car to drive with no engine....halleluja...amen...god is awesome. This man was stoned because he questioned god....halleluja...amen...he will smite those who go up against him. I am biased, biased against stupidity associated with blind faith. Which is exactly why this study was done against psychology. Can you imagine this being done against religion. It would not be embraced, all hell would break lose.

Accept what you want...this is nothing more then my reasonings based on my observations of so called "religious" people.

You think that is a good basis from which to generalise?

I think you are talking about religion at its worst, are you prepared to consider religion at its best?
 

á´…eparted

passages
Joined
Jan 25, 2014
Messages
8,265
You think that is a good basis from which to generalise?

I think you are talking about religion at its worst, are you prepared to consider religion at its best?

The point made is very simple: Would you rather the study of the human mind be based off religon, or science?

The correct answer is science, even if there are flaws.
 

Poki

New member
Joined
Dec 4, 2008
Messages
10,436
MBTI Type
STP
Instinctual Variant
sx/so
You think that is a good basis from which to generalise?

I think you are talking about religion at its worst, are you prepared to consider religion at its best?

I understand religion at it's best. But I still believe it's nothing more then a psychologically driven goal to help people. What I mentioned is actually religions good side as well as its bad. It's what keeps people going and moving and keeping them out of depression. It keeps people positive among all the negativity. It gives them hope that someone cares and loves them eternally. That someone else is watching them. It's a supposed parent or parents watching out for thier kids. It appeased the side of us that doesn't have the love and care that we want, that we need. But it also in the process of providing all this good also pushes people into this cycle as it also breads the weak through crutches instead of solid internal solutions. If religion crashed it something came out proving god without a doubt as false that was not denial the world would go to hell, NT because of lack of god, but because of people's confusion of what to do without the thought of god. All that is good and bad has been followed unquestioned, so they now must question everything because they havnt been taught as to why. They havnt been taught to stand on thier own without god. They have become reliant and dependent.

Sorry, this is my 2 cents if you wanna know. Religion is a psychological tool that is needed for some people to progress this world. Its flaws is that it propagates a circular loop that can and may crash on itself if it is ever to be killed instead of actually focusing on what matters that it preaches. Love, acceptance, self worth, self reliance, all these other areas psychology attempts to handle in a better way then religiin.
 

Stek

New member
Joined
May 22, 2015
Messages
68
MBTI Type
InTJ

Because none of the supernatural stuff in religion is replicable, thus the reliability is non-existent. They are simply claims without any substance.
 

Lark

Active member
Joined
Jun 21, 2009
Messages
29,569
The point made is very simple: Would you rather the study of the human mind be based off religon, or science?

The correct answer is science, even if there are flaws.

I dont believe that science and religion are dichotomous.

The exploitation of every opportunity to reassert that they are and that religion is everywhere and always simply awful has become extremely tedious also.

You all could choose to be more than a broken record repeating the things you heard oft late but at the end of the day you're free to choose and probably happy with the choice you've made.
 

Lark

Active member
Joined
Jun 21, 2009
Messages
29,569
Because none of the supernatural stuff in religion is replicable, thus the reliability is non-existent. They are simply claims without any substance.

I think its strange that it got a mention then, its clearly pressing upon someone's mind that it has to be introduced into a thread which would not appear to be about that topic.

Vicktor Frankl said he thought there was a lot of repression going on in the mind of most atheists he knew, perhaps he was right.
 

á´…eparted

passages
Joined
Jan 25, 2014
Messages
8,265
I dont believe that science and religion are dichotomous.

The exploitation of every opportunity to reassert that they are and that religion is everywhere and always simply awful has become extremely tedious also.

You all could choose to be more than a broken record repeating the things you heard oft late but at the end of the day you're free to choose and probably happy with the choice you've made.

Despite the strong opinions I may have, people can believe what they want to. I am not debating that aspect. What I am debating is the practical real world application of either one.

In the realm of fact, science, and law, they are dichotomous. The second you start trying to merge or compare religion with fact, science, or law though, is the second I stop respecting others differing views. Psychology is a science. Science can be verified and tested. Religion can not, and is nothing more than belief. When it comes to mental ailments, science and medicine is to be trusted to give us the answers on what can be done to remedy things. Religon, can not.
 

Lark

Active member
Joined
Jun 21, 2009
Messages
29,569
I understand religion at it's best. But I still believe it's nothing more then a psychologically driven goal to help people. What I mentioned is actually religions good side as well as its bad. It's what keeps people going and moving and keeping them out of depression. It keeps people positive among all the negativity. It gives them hope that someone cares and loves them eternally. That someone else is watching them. It's a supposed parent or parents watching out for thier kids. It appeased the side of us that doesn't have the love and care that we want, that we need. But it also in the process of providing all this good also pushes people into this cycle as it also breads the weak through crutches instead of solid internal solutions. If religion crashed it something came out proving god without a doubt as false that was not denial the world would go to hell, NT because of lack of god, but because of people's confusion of what to do without the thought of god. All that is good and bad has been followed unquestioned, so they now must question everything because they havnt been taught as to why. They havnt been taught to stand on thier own without god. They have become reliant and dependent.

Sorry, this is my 2 cents if you wanna know. Religion is a psychological tool that is needed for some people to progress this world. Its flaws is that it propagates a circular loop that can and may crash on itself if it is ever to be killed instead of actually focusing on what matters that it preaches. Love, acceptance, self worth, self reliance, all these other areas psychology attempts to handle in a better way then religiin.

I dont agree with that framing of religion at all, its definitely not my experience either, nor does religion fufil those needs for me.

I accept its your two cents, I'll not seek to argue with you or persuade you otherwise since you've obvious got your own reasons for thinking so and if I simply challenge it you're going to discover two counter points for any point I make. However, I would question whether or not you could find that a totally satisfactory answer, maybe you do, I wouldnt. It all reads like a human, all too human, attempt to resolve a difficult question with a simplistic answer in order to give it no further thought and just dismiss it out of hand. That's just my two cents.

Most of the people I've encountered who may have embraced religion like an opiate usually have no recourse to religion at all because they have actual real opiates or booze or meth or something like it to resort to instead and arent inclined towards any substitutes.
 

Stek

New member
Joined
May 22, 2015
Messages
68
MBTI Type
InTJ
I think its strange that it got a mention then, its clearly pressing upon someone's mind that it has to be introduced into a thread which would not appear to be about that topic.

Vicktor Frankl said he thought there was a lot of repression going on in the mind of most atheists he knew, perhaps he was right.

No, it got a mention because it is a widely spread phenomena of thought that is even less credible than psychology as a science. That is why it got mentioned. To draw a line of comparison to something that is relevant in the context of believing, apparently, unverified claims. Because that is what the OP is talking about. If less then half of psychology studies can be replicated, then it is still better than none of them. Which is the case for all supernatural religious claims.

Victor Frankl was probably right when he said that. There is a lot of repression going on in the minds of every human on earth. Including yourself. However, some people are able to realize this. Others are not.
 

Lark

Active member
Joined
Jun 21, 2009
Messages
29,569
No, it got a mention because it is a widely spread phenomena of thought that is even less credible than psychology as a science. That is why it got mentioned. To draw a line of comparison to something that is relevant in the context of believing, apparently, unverified claims. Because that is what the OP is talking about. If less then half of psychology studies can be replicated, then it is still better than none of them. Which is the case for all supernatural religious claims.

Victor Frankl was probably right when he said that. There is a lot of repression going on in the minds of every human on earth. Including yourself. However, some people are able to realize this. Others are not.

Am I to read the implication that you, by virtue of your belief in something other than religion, are able to know your mind, and perhaps the mind of others, including the ability to pick up on repression, and I, by virtue of what you suppose is a belief in religion contra scientific method, am not?

And that does not appear possibly conceited?

I seems to me that your post fails, and fails badly, by your own standards of proof and evidence to know a thing.

NB Frankl was writing in Man's Search of Meaning, so far as I remember, and he was referring to spiritual or religious repression.
 
Last edited:

Stek

New member
Joined
May 22, 2015
Messages
68
MBTI Type
InTJ
Am I to read the implication that you, by virtue of your belief in something other than religion, are able to know your mind, and perhaps the mind of others, including the ability to pick up on repression, and I, by virtue of what you suppose is a belief in religion contra scientific method, am not?

And that does not appear possibly conceited?

I seems to me that your post fails, and fails badly, by your own standards of proof and evidence to know a thing.

NB Frankl was writing in Man's Search of Meaning, so far as I remember, and he was referring to spiritual or religious repression.

What are you talking about? I haven't even implied that you believe in any religion at all. Don't take this personal. This is a conceptual discussion. It is not about you. I don't care what you are able to pick up on.

The point I have made is that supernatural claims in religion cannot be replicated. However, nearly half (according to the OP) of studies in psychology can be replicated. That shows that the reliability of psychological studies is superior to that of supernatural religious claims.

This is all that was implied in my initial post.

Do you have any objections to this point?
 

Poki

New member
Joined
Dec 4, 2008
Messages
10,436
MBTI Type
STP
Instinctual Variant
sx/so
I dont agree with that framing of religion at all, its definitely not my experience either, nor does religion fufil those needs for me.

I accept its your two cents, I'll not seek to argue with you or persuade you otherwise since you've obvious got your own reasons for thinking so and if I simply challenge it you're going to discover two counter points for any point I make. However, I would question whether or not you could find that a totally satisfactory answer, maybe you do, I wouldnt. It all reads like a human, all too human, attempt to resolve a difficult question with a simplistic answer in order to give it no further thought and just dismiss it out of hand. That's just my two cents.

Most of the people I've encountered who may have embraced religion like an opiate usually have no recourse to religion at all because they have actual real opiates or booze or meth or something like it to resort to instead and arent inclined towards any substitutes.

Yes, I don't know you, so I don't know what it is about religion that pulls you in. Also I can give is examples I see in others. They all don't apply to everyone as we are all different and have different needs and wants. IMHO religion as it is today is not a satisfactory answer for self help that psychology attempts to accomplish as it destroys the mind in order to bring the soul up. The best solution IMHO would bring both up and find a balance, not supress one to raise the other. It is a solution, but I prefer to have hope that their is a better solution.
 
Top