Reproducibility crisis in Science

[Maou]

Most scientists 'can't replicate studies by their peers' - BBC News

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests."

This is what happens when you don't review, and too readily share information, that hasn't been reproduced more than once, or basing your entire premise on one old possibly faulty study.

And people thought I was crazy for doubting "science" a lot of the time (especially with popular fad topics). More like I don't trust some 40 year old study that has never been done again, and had vague results at best.

 

[The Cat]

Ive never thought you crazy for doubting science, I think you're crazy for entirely different reasons.

 

[Lark]

Some research findings should invite skepticism but they shouldnt invite cynicism, or wholesale distrust, and I am of the view that even when overturned most studies and research findings are learning opportunities, even if its how to better design a study, how not to design a study even.

If findings are fabricated or influenced some how that can be significant all by itself because it could tell you something about the researchers, the times they were performing the research in, the participants etc. etc.

Even the Stanford Prison Experiment, the Obedience to Authority tests, the separated Twins and Triplets studies all are excellent cases for studying in terms of ethics, harm principles and consequentialism. They were all bad studies with (at least in the first two instances) poor findings.

Although, that said, I'd say its safe to find a few cautionary tales about science since it is one of those body's of knowledge which has almost religious propensity for abuse and control etc. What I consider important to remember personally are the histories of things like Phrenology, Palmistry, Physiology and Criminal offending etc. Although caution about science, particularly "mad science" which there was at a time more concern about, shouldnt be about wholesale dumping on it. I'm grateful particularly for medicine and medical science.

 

[Maou]

Ive never thought you crazy for doubting science, I think you're crazy for entirely different reasons.

Am I at least the good or bad kind of crazy, my fellow madman.

 

[Tellenbach]

The entire "cholesterol and saturated foods are bad for you" theory is based on a faulty study (a really bad case of cherrypicking data); many tens of millions of people have died because the government discouraged the consumption of meat and encouraged the consumption of processed carbohydrates. It would be a great idea to review all current policies that are based on "scientific" studies.

 

[Vendrah]

It depends in which science you are talking about. The article points one specific study in medicine related to cancer studies. You cant say that every other part of the science is in reproducibility cryisis because of one specific area of medicine. Of course that the title suggests that, but that just BBC working out (its a mild manipulation) to get more clicks. Actually, there is a quote inside that article: "It's about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about."; BBC actually did exactly this in the article, they guided the reader to a fake conclusion that the whole science is in a reproducibility crisis by considering five studies in an specific science area (we know that there are thousands or milions studies, depending on the time frame you consider). And, of course, they did it in a reticent way (even mild manipulation is supposed to go unnoticed). This doesnt mean that there is not a reproducibility crisis, but it doesnt mean that there is one either.

Physics and chemistry are more rigorous and precise than psychology (these are the two extremes I think). I dont think reproducibility in physics or chemistry is "in crisis", although a few of their experiments are incredibly expensive these days (and might require a billionarie budget to be done). Psychology experiments are expected to have more variation and some of them does not have a list of participants that is truly random. Some of them have a list of participants that are inside the same culture or country, one specific psychology experiment in one country can lead to conclusions that appears to be related to psychology but could be culture-related. And sometimes these two things can even mix.

 

[Coriolis]

Most scientists 'can't replicate studies by their peers' - BBC News

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests."

This is what happens when you don't review, and too readily share information, that hasn't been reproduced more than once, or basing your entire premise on one old possibly faulty study.

And people thought I was crazy for doubting "science" a lot of the time (especially with popular fad topics). More like I don't trust some 40 year old study that has never been done again, and had vague results at best.

Why is this in the politics subforum instead of science/technology?

 

[Maou]

Why is this in the politics subforum instead of science/technology?

Because its been a talking point in politics quite a bit.

 

[Coriolis]

Because its been a talking point in politics quite a bit.

How so? The concerns raised in the article seem to be within the scientific profession, and point to the need for better self-policing. Could you elaborate on the political aspects of that, if you want that to be the focus of the thread? I find that politics and science mixes poorly in any case. Once scientists start considering the political angles of what they are doing, it just goes downhill from there.

 

[rav3n]

Ummm...that's what peer reviewing is all about and why it's important to ensure that whatever study you cite, is housed in a peer reviewed journal.

 

[ceecee]

Because its been a talking point in politics quite a bit.

Conservatives made it political, that doesn't mean it actually is.

 

[The Cat]

Us vs us will always lead to inevitable difficulties in reproducing. Just the way They like it.

 

[Tellenbach]

Earlier in the year, there was a study that said drinking lots of coffee was bad for you and then about a month later, there's another study that says drinking coffee is good for you. Maybe one study used 7/11 coffee and the other used Starbucks coffee?

 

[ceecee]

It depends in which science you are talking about. The article points one specific study in medicine related to cancer studies. You cant say that every other part of the science is in reproducibility cryisis because of one specific area of medicine. Of course that the title suggests that, but that just BBC working out (its a mild manipulation) to get more clicks. Actually, there is a quote inside that article: "It's about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about."; BBC actually did exactly this in the article, they guided the reader to a fake conclusion that the whole science is in a reproducibility crisis by considering five studies in an specific science area (we know that there are thousands or milions studies, depending on the time frame you consider). And, of course, they did it in a reticent way (even mild manipulation is supposed to go unnoticed). This doesnt mean that there is not a reproducibility crisis, but it doesnt mean that there is one either.

Physics and chemistry are more rigorous and precise than psychology (these are the two extremes I think). I dont think reproducibility in physics or chemistry is "in crisis", although a few of their experiments are incredibly expensive these days (and might require a billionarie budget to be done). Psychology experiments are expected to have more variation and some of them does not have a list of participants that is truly random. Some of them have a list of participants that are inside the same culture or country, one specific psychology experiment in one country can lead to conclusions that appears to be related to psychology but could be culture-related. And sometimes these two things can even mix.

From what I gathered after reading, this is mostly a funding body problem, which makes sense. Look at what just happened at MIT for example. So this is less about reproducing and more about the cherry picking of data to get the most bang for the buck. This isn't talking about junk science or how we went from here...

Bush Had a Lasting Impact on Climate and Air Policy - Scientific American

to here

Climate explained: Why are climate change skeptics often right-wing conservatives?

It's more like..

Quillette, Ben Shapiro, and the Myth of Conservative ‘Facts’

and this

The age of cherry-picking | PolitiFact

Oh and that the heads of every US agency that oversees industry is headed by lobbyists for that the industry, including research but this isn't Trump or anything new. W really got that ball rolling.

 

[Coriolis]

Earlier in the year, there was a study that said drinking lots of coffee was bad for you and then about a month later, there's another study that says drinking coffee is good for you. Maybe one study used 7/11 coffee and the other used Starbucks coffee?

I assume you are familiar with the expression: too much of a good thing. Real scientific publications will make sure to place quantitative bounds on "lots", plus add in mitigating or complicating factors.

 

[Tellenbach]

More than 4 cups of coffee linked to early death.

A Mayo Clinic partnered study found that men who drank more than four 8 fl.oz. cups of coffee had a 21% increase in all-cause mortality.

Another study showed that those who consume 6+ coffees per day have a greater risk of developing heart disease.

Coffee not as bad for heart and circulatory system as previously thought

The research from Queen Mary University of London has shown that drinking coffee, including in people who drink up to 25 cups a day, is not associated with having stiffer arteries.

"There are several conflicting studies saying different things about coffee, and it can be difficult to filter what we should believe and what we shouldn't. This research will hopefully put some of the media reports in perspective, as it rules out one of the potential detrimental effects of coffee on our arteries."

In general, I distrust research that warns against eating delicious food items. I also think any risk should be weighed against the pleasure/enjoyment that it gives. I'm sure we'll be hearing about the dangers of seared scallops or braised beef in the future from do-gooders.

 

[Vendrah]

From what I gathered after reading, this is mostly a funding body problem, which makes sense. Look at what just happened at MIT for example. So this is less about reproducing and more about the cherry picking of data to get the most bang for the buck. This isn't talking about junk science or how we went from here...

Bush Had a Lasting Impact on Climate and Air Policy - Scientific American

to here

Climate explained: Why are climate change skeptics often right-wing conservatives?

It's more like..

Quillette, Ben Shapiro, and the Myth of Conservative ‘Facts’

and this

The age of cherry-picking | PolitiFact

Oh and that the heads of every US agency that oversees industry is headed by lobbyists for that the industry, including research but this isn't Trump or anything new. W really got that ball rolling.

I had a read in these articles although I am not from the US.
Manipulation like what you call "cherry picking" is very common and spread in my country several media (although I dont know face-to-face anyone who noticed them but myself; I already tried to give enlightment but it seems impossible, people will even cry for their favorite political messiah), like in WhatsApp, TV Shows, local you tubers and sites, etc... Here, this tatic is present and used on the left-wing and right-wing at the same time (the right-wing here is the worse since they have the best tricks). The cherry-picking, when you catch its principle, has several uses for manipulation purposes, like inducing racism (without the governemnt being able to argue against), changing the watchers perceptions (and what they perceive as realistic or usual or even as 'good' or 'bad') in several areas, like aesthetics (notion of what is pretty and what is ugly), politics, statistical manipulation (I only saw it to support a political point so far) and these are the ones I remember right now. Im not going to be more specific on the subject because I am already derailing from the topic. And about these people who despise evidence for their political ground... Trust me, it can get worse, in the place I live in, like half people or more only believe studys and data which supports their Messiah and ignore the others ones their Messiah doesnt like... Its depressing, I am trying to taught myself to not care so I wont get mad... And I am talking about two Messiahs actually, but whatever.

Back to topic, your examples are more political cherry picking, from media, than actually scientifical papers doing cherry picking. However I know that there could (or maybe 'should' is more appropriate) be some studies with "lobbyists" interested on manipulating the results to support their interests (and this is a good point for the governemnt to keep a minimal founding in science) but I dont see it coming endemic... yet.

 

[ceecee]

I had a read in these articles although I am not from the US.
Manipulation like what you call "cherry picking" is very common and spread in my country several media (although I dont know face-to-face anyone who noticed them but myself; I already tried to give enlightment but it seems impossible, people will even cry for their favorite political messiah), like in WhatsApp, TV Shows, local you tubers and sites, etc... Here, this tatic is present and used on the left-wing and right-wing at the same time (the right-wing here is the worse since they have the best tricks). The cherry-picking, when you catch its principle, has several uses for manipulation purposes, like inducing racism (without the governemnt being able to argue against), changing the watchers perceptions (and what they perceive as realistic or usual or even as 'good' or 'bad') in several areas, like aesthetics (notion of what is pretty and what is ugly), politics, statistical manipulation (I only saw it to support a political point so far) and these are the ones I remember right now. Im not going to be more specific on the subject because I am already derailing from the topic. And about these people who despise evidence for their political ground... Trust me, it can get worse, in the place I am live in like half people only believe studys and data which supports their Messiah and ignore the others one their Messiah doesnt like... Its depressing, I am trying to taught myself to not care... And I am talking about two Messiahs actually, but whatever.

Back to topic, your examples are more political cherry picking, from media, than actually scientifical papers doing cherry picking. However I know that there could (or maybe 'should' is more appropriate) be some studies with "lobbyists" interested on manipulating the results to support their interests (and this is a good point for the governemnt to keep a minimal founding in science) but I dont see it coming endemic... yet.

What I mean is that the topics of research are often "cherry picked", to get the most funding and what it says in journals reflect the desires of that funding. And it's not as though the results can't be reproduced at all. From the OP article...

The problem, it turned out, was not with Marcus Munafo's science, but with the way the scientific literature had been "tidied up" to present a much clearer, more robust outcome.

"What we see in the published literature is a highly curated version of what's actually happened," he says.

"The trouble is that gives you a rose-tinted view of the evidence because the results that get published tend to be the most interesting, the most exciting, novel, eye-catching, unexpected results.

"What I think of as high-risk, high-return results."

The reproducibility difficulties are not about fraud, according to Dame Ottoline Leyser, director of the Sainsbury Laboratory at the University of Cambridge.

That would be relatively easy to stamp out. Instead, she says: "It's about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about."

The bolded is the point I am making. This is NOT about "doubting science" or some other bullshit. This is a problem due to lack of funding minus strings and researchers being able to research and reproduce no matter what the actual topic is.

 

[Vendrah]

What I mean is that the topics of research are often "cherry picked", to get the most funding and what it says in journals reflect the desires of that funding. And it's not as though the results can't be reproduced at all. From the OP article...





The bolded is the point I am making. This is NOT about "doubting science" or some other bullshit. This is a problem due to lack of funding minus strings and researchers being able to research and reproduce no matter what the actual topic is.

I got your point.
Thats why at least some part of research funding should come from goverment, because public funding is more likely to have a neutrality than private ones.
However, the OP article is more opinative than really "statitive" and the reason for that is in my first post (he concludes that based on 6 cancer studies). We cant really say thats happening, as I said, it could be and it can turn to be endemic (or not).

 

[The Cat]

It seems like the right and left both like to claim they're the rational ones devoted to science, but both tend to cherry pick or ignore scientific findings when they don't align with their worldviews.

Right wingers tend to ignore or downplay climate change, or will argue it's not really man-made, so why bother worrying about it? Left wingers tend to ignore the biology behind sex, and in some circles the dangers of obesity (i.e. the fat acceptance movement).

Both sides are guilty of willfully ignoring and cherry picking.

Call a side in the air; it's all the same coin.