• You are currently viewing our forum as a guest, which gives you limited access to view most discussions and access our other features. By joining our free community, you will have access to additional post topics, communicate privately with other members (PM), view blogs, respond to polls, upload content, and access many other special features. Registration is fast, simple and absolutely free, so please join our community today! Just click here to register. You should turn your Ad Blocker off for this site or certain features may not work properly. If you have any problems with the registration process or your account login, please contact us by clicking here.

Reproducibility crisis in Science

Maou

Mythos
Joined
Jun 20, 2018
Messages
6,117
MBTI Type
INTP
Enneagram
5w6
Instinctual Variant
sx/sp
Most scientists 'can't replicate studies by their peers' - BBC News

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests."

This is what happens when you don't review, and too readily share information, that hasn't been reproduced more than once, or basing your entire premise on one old possibly faulty study.

And people thought I was crazy for doubting "science" a lot of the time (especially with popular fad topics). More like I don't trust some 40 year old study that has never been done again, and had vague results at best.
 

The Cat

Just a Cat who hangs out at the Crossroads
Staff member
Joined
Oct 15, 2016
Messages
23,552
Ive never thought you crazy for doubting science, I think you're crazy for entirely different reasons.:wubbie:
 

Lark

Active member
Joined
Jun 21, 2009
Messages
29,569
Some research findings should invite skepticism but they shouldnt invite cynicism, or wholesale distrust, and I am of the view that even when overturned most studies and research findings are learning opportunities, even if its how to better design a study, how not to design a study even.

If findings are fabricated or influenced some how that can be significant all by itself because it could tell you something about the researchers, the times they were performing the research in, the participants etc. etc.

Even the Stanford Prison Experiment, the Obedience to Authority tests, the separated Twins and Triplets studies all are excellent cases for studying in terms of ethics, harm principles and consequentialism. They were all bad studies with (at least in the first two instances) poor findings.

Although, that said, I'd say its safe to find a few cautionary tales about science since it is one of those body's of knowledge which has almost religious propensity for abuse and control etc. What I consider important to remember personally are the histories of things like Phrenology, Palmistry, Physiology and Criminal offending etc. Although caution about science, particularly "mad science" which there was at a time more concern about, shouldnt be about wholesale dumping on it. I'm grateful particularly for medicine and medical science.
 

Tellenbach

in dreamland
Joined
Oct 27, 2013
Messages
6,088
MBTI Type
ISTJ
Enneagram
6w5
The entire "cholesterol and saturated foods are bad for you" theory is based on a faulty study (a really bad case of cherrypicking data); many tens of millions of people have died because the government discouraged the consumption of meat and encouraged the consumption of processed carbohydrates. It would be a great idea to review all current policies that are based on "scientific" studies.
 

Vendrah

Well-known member
Joined
Mar 26, 2017
Messages
1,947
MBTI Type
NP
Enneagram
952
It depends in which science you are talking about. The article points one specific study in medicine related to cancer studies. You cant say that every other part of the science is in reproducibility cryisis because of one specific area of medicine. Of course that the title suggests that, but that just BBC working out (its a mild manipulation) to get more clicks. Actually, there is a quote inside that article: "It's about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about."; BBC actually did exactly this in the article, they guided the reader to a fake conclusion that the whole science is in a reproducibility crisis by considering five studies in an specific science area (we know that there are thousands or milions studies, depending on the time frame you consider). And, of course, they did it in a reticent way (even mild manipulation is supposed to go unnoticed). This doesnt mean that there is not a reproducibility crisis, but it doesnt mean that there is one either.

Physics and chemistry are more rigorous and precise than psychology (these are the two extremes I think). I dont think reproducibility in physics or chemistry is "in crisis", although a few of their experiments are incredibly expensive these days (and might require a billionarie budget to be done). Psychology experiments are expected to have more variation and some of them does not have a list of participants that is truly random. Some of them have a list of participants that are inside the same culture or country, one specific psychology experiment in one country can lead to conclusions that appears to be related to psychology but could be culture-related. And sometimes these two things can even mix.
 

Coriolis

Si vis pacem, para bellum
Staff member
Joined
Apr 18, 2010
Messages
27,230
MBTI Type
INTJ
Enneagram
5w6
Instinctual Variant
sp/sx
Most scientists 'can't replicate studies by their peers' - BBC News

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests."

This is what happens when you don't review, and too readily share information, that hasn't been reproduced more than once, or basing your entire premise on one old possibly faulty study.

And people thought I was crazy for doubting "science" a lot of the time (especially with popular fad topics). More like I don't trust some 40 year old study that has never been done again, and had vague results at best.
Why is this in the politics subforum instead of science/technology?
 

Coriolis

Si vis pacem, para bellum
Staff member
Joined
Apr 18, 2010
Messages
27,230
MBTI Type
INTJ
Enneagram
5w6
Instinctual Variant
sp/sx
Because its been a talking point in politics quite a bit.
How so? The concerns raised in the article seem to be within the scientific profession, and point to the need for better self-policing. Could you elaborate on the political aspects of that, if you want that to be the focus of the thread? I find that politics and science mixes poorly in any case. Once scientists start considering the political angles of what they are doing, it just goes downhill from there.
 

rav3n

.
Joined
Aug 6, 2010
Messages
11,655
Ummm...that's what peer reviewing is all about and why it's important to ensure that whatever study you cite, is housed in a peer reviewed journal.
 

The Cat

Just a Cat who hangs out at the Crossroads
Staff member
Joined
Oct 15, 2016
Messages
23,552
Us vs us will always lead to inevitable difficulties in reproducing. Just the way They like it.
 

Tellenbach

in dreamland
Joined
Oct 27, 2013
Messages
6,088
MBTI Type
ISTJ
Enneagram
6w5
Earlier in the year, there was a study that said drinking lots of coffee was bad for you and then about a month later, there's another study that says drinking coffee is good for you. Maybe one study used 7/11 coffee and the other used Starbucks coffee?
 

ceecee

Coolatta® Enjoyer
Joined
Apr 22, 2008
Messages
15,908
MBTI Type
INTJ
Enneagram
8w9
It depends in which science you are talking about. The article points one specific study in medicine related to cancer studies. You cant say that every other part of the science is in reproducibility cryisis because of one specific area of medicine. Of course that the title suggests that, but that just BBC working out (its a mild manipulation) to get more clicks. Actually, there is a quote inside that article: "It's about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about."; BBC actually did exactly this in the article, they guided the reader to a fake conclusion that the whole science is in a reproducibility crisis by considering five studies in an specific science area (we know that there are thousands or milions studies, depending on the time frame you consider). And, of course, they did it in a reticent way (even mild manipulation is supposed to go unnoticed). This doesnt mean that there is not a reproducibility crisis, but it doesnt mean that there is one either.

Physics and chemistry are more rigorous and precise than psychology (these are the two extremes I think). I dont think reproducibility in physics or chemistry is "in crisis", although a few of their experiments are incredibly expensive these days (and might require a billionarie budget to be done). Psychology experiments are expected to have more variation and some of them does not have a list of participants that is truly random. Some of them have a list of participants that are inside the same culture or country, one specific psychology experiment in one country can lead to conclusions that appears to be related to psychology but could be culture-related. And sometimes these two things can even mix.

From what I gathered after reading, this is mostly a funding body problem, which makes sense. Look at what just happened at MIT for example. So this is less about reproducing and more about the cherry picking of data to get the most bang for the buck. This isn't talking about junk science or how we went from here...

Bush Had a Lasting Impact on Climate and Air Policy - Scientific American

to here

Climate explained: Why are climate change skeptics often right-wing conservatives?

It's more like..

Quillette, Ben Shapiro, and the Myth of Conservative ‘Facts’

and this

The age of cherry-picking | PolitiFact

Oh and that the heads of every US agency that oversees industry is headed by lobbyists for that the industry, including research but this isn't Trump or anything new. W really got that ball rolling.
 

Coriolis

Si vis pacem, para bellum
Staff member
Joined
Apr 18, 2010
Messages
27,230
MBTI Type
INTJ
Enneagram
5w6
Instinctual Variant
sp/sx
Earlier in the year, there was a study that said drinking lots of coffee was bad for you and then about a month later, there's another study that says drinking coffee is good for you. Maybe one study used 7/11 coffee and the other used Starbucks coffee?
I assume you are familiar with the expression: too much of a good thing. Real scientific publications will make sure to place quantitative bounds on "lots", plus add in mitigating or complicating factors.
 

Tellenbach

in dreamland
Joined
Oct 27, 2013
Messages
6,088
MBTI Type
ISTJ
Enneagram
6w5
More than 4 cups of coffee linked to early death.

A Mayo Clinic partnered study found that men who drank more than four 8 fl.oz. cups of coffee had a 21% increase in all-cause mortality.

Another study showed that those who consume 6+ coffees per day have a greater risk of developing heart disease.

Coffee not as bad for heart and circulatory system as previously thought

The research from Queen Mary University of London has shown that drinking coffee, including in people who drink up to 25 cups a day, is not associated with having stiffer arteries.

"There are several conflicting studies saying different things about coffee, and it can be difficult to filter what we should believe and what we shouldn't. This research will hopefully put some of the media reports in perspective, as it rules out one of the potential detrimental effects of coffee on our arteries."

In general, I distrust research that warns against eating delicious food items. I also think any risk should be weighed against the pleasure/enjoyment that it gives. I'm sure we'll be hearing about the dangers of seared scallops or braised beef in the future from do-gooders. :D
 

Vendrah

Well-known member
Joined
Mar 26, 2017
Messages
1,947
MBTI Type
NP
Enneagram
952
From what I gathered after reading, this is mostly a funding body problem, which makes sense. Look at what just happened at MIT for example. So this is less about reproducing and more about the cherry picking of data to get the most bang for the buck. This isn't talking about junk science or how we went from here...

Bush Had a Lasting Impact on Climate and Air Policy - Scientific American

to here

Climate explained: Why are climate change skeptics often right-wing conservatives?

It's more like..

Quillette, Ben Shapiro, and the Myth of Conservative ‘Facts’

and this

The age of cherry-picking | PolitiFact

Oh and that the heads of every US agency that oversees industry is headed by lobbyists for that the industry, including research but this isn't Trump or anything new. W really got that ball rolling.

I had a read in these articles although I am not from the US.
Manipulation like what you call "cherry picking" is very common and spread in my country several media (although I dont know face-to-face anyone who noticed them but myself; I already tried to give enlightment but it seems impossible, people will even cry for their favorite political messiah), like in WhatsApp, TV Shows, local you tubers and sites, etc... Here, this tatic is present and used on the left-wing and right-wing at the same time (the right-wing here is the worse since they have the best tricks). The cherry-picking, when you catch its principle, has several uses for manipulation purposes, like inducing racism (without the governemnt being able to argue against), changing the watchers perceptions (and what they perceive as realistic or usual or even as 'good' or 'bad') in several areas, like aesthetics (notion of what is pretty and what is ugly), politics, statistical manipulation (I only saw it to support a political point so far) and these are the ones I remember right now. Im not going to be more specific on the subject because I am already derailing from the topic. And about these people who despise evidence for their political ground... Trust me, it can get worse, in the place I live in, like half people or more only believe studys and data which supports their Messiah and ignore the others ones their Messiah doesnt like... Its depressing, I am trying to taught myself to not care so I wont get mad... And I am talking about two Messiahs actually, but whatever.

Back to topic, your examples are more political cherry picking, from media, than actually scientifical papers doing cherry picking. However I know that there could (or maybe 'should' is more appropriate) be some studies with "lobbyists" interested on manipulating the results to support their interests (and this is a good point for the governemnt to keep a minimal founding in science) but I dont see it coming endemic... yet.
 

ceecee

Coolatta® Enjoyer
Joined
Apr 22, 2008
Messages
15,908
MBTI Type
INTJ
Enneagram
8w9
I had a read in these articles although I am not from the US.
Manipulation like what you call "cherry picking" is very common and spread in my country several media (although I dont know face-to-face anyone who noticed them but myself; I already tried to give enlightment but it seems impossible, people will even cry for their favorite political messiah), like in WhatsApp, TV Shows, local you tubers and sites, etc... Here, this tatic is present and used on the left-wing and right-wing at the same time (the right-wing here is the worse since they have the best tricks). The cherry-picking, when you catch its principle, has several uses for manipulation purposes, like inducing racism (without the governemnt being able to argue against), changing the watchers perceptions (and what they perceive as realistic or usual or even as 'good' or 'bad') in several areas, like aesthetics (notion of what is pretty and what is ugly), politics, statistical manipulation (I only saw it to support a political point so far) and these are the ones I remember right now. Im not going to be more specific on the subject because I am already derailing from the topic. And about these people who despise evidence for their political ground... Trust me, it can get worse, in the place I am live in like half people only believe studys and data which supports their Messiah and ignore the others one their Messiah doesnt like... Its depressing, I am trying to taught myself to not care... And I am talking about two Messiahs actually, but whatever.

Back to topic, your examples are more political cherry picking, from media, than actually scientifical papers doing cherry picking. However I know that there could (or maybe 'should' is more appropriate) be some studies with "lobbyists" interested on manipulating the results to support their interests (and this is a good point for the governemnt to keep a minimal founding in science) but I dont see it coming endemic... yet.

What I mean is that the topics of research are often "cherry picked", to get the most funding and what it says in journals reflect the desires of that funding. And it's not as though the results can't be reproduced at all. From the OP article...

The problem, it turned out, was not with Marcus Munafo's science, but with the way the scientific literature had been "tidied up" to present a much clearer, more robust outcome.

"What we see in the published literature is a highly curated version of what's actually happened," he says.

"The trouble is that gives you a rose-tinted view of the evidence because the results that get published tend to be the most interesting, the most exciting, novel, eye-catching, unexpected results.

"What I think of as high-risk, high-return results."

The reproducibility difficulties are not about fraud, according to Dame Ottoline Leyser, director of the Sainsbury Laboratory at the University of Cambridge.

That would be relatively easy to stamp out. Instead, she says: "It's about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about."

The bolded is the point I am making. This is NOT about "doubting science" or some other bullshit. This is a problem due to lack of funding minus strings and researchers being able to research and reproduce no matter what the actual topic is.
 

Vendrah

Well-known member
Joined
Mar 26, 2017
Messages
1,947
MBTI Type
NP
Enneagram
952
What I mean is that the topics of research are often "cherry picked", to get the most funding and what it says in journals reflect the desires of that funding. And it's not as though the results can't be reproduced at all. From the OP article...





The bolded is the point I am making. This is NOT about "doubting science" or some other bullshit. This is a problem due to lack of funding minus strings and researchers being able to research and reproduce no matter what the actual topic is.

I got your point.
Thats why at least some part of research funding should come from goverment, because public funding is more likely to have a neutrality than private ones.
However, the OP article is more opinative than really "statitive" and the reason for that is in my first post (he concludes that based on 6 cancer studies). We cant really say thats happening, as I said, it could be and it can turn to be endemic (or not).
 

The Cat

Just a Cat who hangs out at the Crossroads
Staff member
Joined
Oct 15, 2016
Messages
23,552
It seems like the right and left both like to claim they're the rational ones devoted to science, but both tend to cherry pick or ignore scientific findings when they don't align with their worldviews.

Right wingers tend to ignore or downplay climate change, or will argue it's not really man-made, so why bother worrying about it? Left wingers tend to ignore the biology behind sex, and in some circles the dangers of obesity (i.e. the fat acceptance movement).

Both sides are guilty of willfully ignoring and cherry picking.


Call a side in the air; it's all the same coin.
bigstock-Coin-Flip-5807921.jpg
 
Top