• You are currently viewing our forum as a guest, which gives you limited access to view most discussions and access our other features. By joining our free community, you will have access to additional post topics, communicate privately with other members (PM), view blogs, respond to polls, upload content, and access many other special features. Registration is fast, simple and absolutely free, so please join our community today! Just click here to register. You should turn your Ad Blocker off for this site or certain features may not work properly. If you have any problems with the registration process or your account login, please contact us by clicking here.

"Is there something wrong with the scientific method?"

LeafAndSky

New member
Joined
Nov 12, 2009
Messages
307
MBTI Type
ISFP
Interesting article here:

http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer?currentPage=all

"The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe."
 

Blank

.
Joined
Mar 10, 2009
Messages
1,201
MBTI Type
INTP
Enneagram
5w6
This is kind of a difficult subject, and it's incredibly broad in its scope. I've heard discourse which critiques the scientific method because it isn't holistic enough--which has merit and influences much of the personal bias in experiments; however, the scientific method in itself isn't flawed. It's implementation by biased and imperfect beings (humans) often skews results, as shown in the article.

As the article states, one person's quest to disprove a data set:
"'Unfortunately, I couldn’t find the effect. But the worst part was that when I submitted these null results I had difficulty getting them published. The journals only wanted confirming data. It was too exciting an idea to disprove, at least back then.'"

For the most part, it's not the scientific method that is flawed--it is we who are flawed.
 

Randomnity

insert random title here
Joined
May 8, 2007
Messages
9,485
MBTI Type
ISTP
Enneagram
6w5
Instinctual Variant
sp/sx
Yes, I've heard of that before. There's so many potential biases from so many people that can interfere with "good science".

The status quo is hard to fight. Though, if many people have gotten result A and you don't, the onus is on you to show stronger than usual support that result A is wrong. There are also other biases, like in (hopefully rare) cases since the journals are peer-reviewed, there's a possibility of getting a reviewer who is working on the same stuff you are...and it's happened sometimes that they'll reject good papers because they don't want them publishing first.

Another bias is when pharmaceutical companies get involved - if you're funded by them, it's very difficult to publish results that say the drug doesn't work (it could even be in your contract). There've been cases where they might do the experiment 5 times, pick the 2 times the drug worked, and publish only those. Or if the effect of the drug starts to"decline", you can have people publishing only the initial experiments where it worked (thankfully, this is much much harder to fudge when real people are being tested, since there are many more rules and more scrutiny, but it can still happen). Actually, that happens even outside of pharmaceuticals - even basic scientists can be tempted to do that deliberately to get nice looking results (but it isn't science, obviously, and isn't much better than making up numbers - which also happens on occasion). IMO it can be an unconscious process too, where you decide that an experiment was flawed because it "doesn't make sense". It's hard not to fall into that when it makes everything easier to explain and understand (and you may even be right - it may have been flawed without you realizing it! it's easy to accidentally put too much reagent or something without realizing it)

It's tempting for people to push results to be a little more favourable in whatever way, whether it's manipulating statistics, selectively choosing experiments, creatively excluding "outliers", excluding "negative" results, etc. It's not immediately obvious that doing this is very harmful for the scientific community. It's basically poisoning the knowledge base.

I think it's also a significant problem that journals are less likely to publish the less-exciting "negative" results (i.e. results saying that there's no difference between treatments and your theory is wrong, vs. ones supporting it). Then you have people wasting time doing the same experiments because the information wasn't available to say that it's a waste of time. Or worse, doing experiments with the assumption that theory A is true, because 5 studies were published supporting it and you can't see the 50 unpublished studies where there wasn't a correlation. There is actually a "Journal of negative results" but I'm not sure publishing there would be good for your reputation!

So yeah, it's not so much the scientific method at fault, but human bias. As rational and objective as we (scientists especially) try to be and think we are, our biases and weaknesses often show up.

I read the paper mentioned in the article, a few months ago (Why Most Published Research Findings Are False). It's a good read, I recommend it.
 

Randomnity

insert random title here
Joined
May 8, 2007
Messages
9,485
MBTI Type
ISTP
Enneagram
6w5
Instinctual Variant
sp/sx
There's also the issue of people by nature seeing patterns in random noise, which can happen in science too. Though there's a lot of rules like statistics to keep that subjectivity to a minimum, it's very possible to have results show a significant difference between groups purely by random chance (The standard is to say results are "different" when there's a 5% chance that the difference is due purely to chance - which is actually a pretty big chance!). That's why repeating experiments and getting similar results, particularly by different labs, is so very important.

There's also the issue that results may be different statistically, but the difference isn't relevant biologically.

Like hypothetically, if a drug is absorbed 50% more effectively in this new formulation, that's an exciting result! But it could be that this difference doesn't translate into a change in how the patients respond to the drug in any way, for whatever reason.
 

erm

Permabanned
Joined
Jun 19, 2007
Messages
1,652
MBTI Type
INFP
Enneagram
5
Well there's lots wrong with the scientific method.

For one it is built upon unproven assumptions. It, being empirical, shares all of empiricism's flaws as well (e.g. "we've controlled all the variables and are still getting dodgy results!" as is hinted at in the article). Hypothesis building and interpretation of data rely heavily on unpredictable creativity. Finally it's very undefined and relies solely upon certain people to decide when something becomes "true".

Outside of the method itself there are some other flaws: Human error and bias (machine error etc), data abstraction being unreliable, and so on.

There are better truth finding methods out there, most famously formal logic, but everyday empirical methods and rational ones outside of science are underestimated in how effective they can be (they are responsible for the majority of technology).

Alas, science often gets separated and idealised from the humans who practice it (almost becoming a synonym for "truth"), treated as a singular entity (reification fallacy), and some even include technology and mathematics within its definition (probably because of science fiction). It may well be why you get those terrible "science" articles everywhere, which betray the scientific method entirely, and yet are simply taken as fact by many.

http://www.guardian.co.uk/science/the-lay-scientist/2010/sep/24/1

"Hard" science tends to adopt a lot from formal logic, keeping empiricism to a minimum, but treating "soft" science as a gospel of truth can have more extreme consequences.

http://blogs.discovermagazine.com/d...classify-happiness-as-a-psychiatric-disorder/ :D
 

LeafAndSky

New member
Joined
Nov 12, 2009
Messages
307
MBTI Type
ISFP
The possibility of "invisible variables we don't understand" commonly influencing results makes my head swim. Add to that the reality that negative findings are likely to go unpublished, plus the difficulty of getting unbiased results because of inherent human psychological or behavioral tendencies . . . and of course the frequency of ethics problems (falsifying results) . . . I feel like giving up reading science altogether.

It was bad enough when as a 'lay person' I could read about research and see that frequently the interpretation given was only one interpretation among many, and the researcher offering it apparently had no idea there were other ways to look at the facts discovered by the research. But now it seems that even the facts are suspect.

A friend reminded me, though, and my son also, that science is theories, and practical results have come from our theories, so don't throw the whole endeavor out just yet. :)

BTW, thank you for the links, erm. Fun.
 

Randomnity

insert random title here
Joined
May 8, 2007
Messages
9,485
MBTI Type
ISTP
Enneagram
6w5
Instinctual Variant
sp/sx
It's not like the flaws make science worthless (there's been plenty of useful innovation in biomed, biotech, engineering, etc etc recently), you just have to keep in mind that individual studies (and theories) can easily be misleading. Hence, pop science often reporting that we've cured cancer and everything else, based on one study. If you have many studies from different research groups reporting the same thing, it's pretty unlikely that they're all in error (though the explanations may be wrong).

It's hard for laypeople to understand a topic of research when often so much is unknown even by the scientists. The jargon barrier doesn't help, either.
 

BlueGray

New member
Joined
Oct 7, 2009
Messages
474
MBTI Type
INTP
Enneagram
5
I really dislike that last paragraph. It's an article about people abusing chance; and they end by saying that proven information isn't true. The entire point of the article was that it wasn't proven. If it truly was proven it is true by definition. The fact that the language of it sounds so similar to what is used when arguing for religion probably increase how much I dislike it.
 

Octarine

The Eighth Colour
Joined
Oct 14, 2007
Messages
1,351
MBTI Type
Aeon
Enneagram
10w
Instinctual Variant
so
There is no well defined "method", but there are lots of limitations in the way science is currently practised. The big ones for me are:

The topics which are actually researched - funding is complicated, but based on what is politically (due to outside funding) and socially (since individuals need to make a choice) acceptable. The funding for certain medical disorders for example is way out of wack with the expected benefits to patients.
Likewise, it is difficult to find funding for controversial studies.
Secondly, there are many biases to do with the communication of scientific knowledge.

Even on the low level, there are many flaws to do with publishing and the peer review system.
I don't have the time to go into detail, but here is a few of my bookmarks (actually, I lost all my old ones recently, so..)

http://www.the-scientist.com/templa...year=2010&page=36&month=8&o_url=2010/8/1/36/1 "I Hate Your Paper"
http://www.plosone.org/article/info:doi/10.1371/journal.pone.0010271 "Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data"
http://www.guardian.co.uk/commentisfree/2008/feb/27/pharmaceuticalindustry "A quick fix would stop drug firms bending the truth"

I think that scientists should register there experiments and publish their hypothesis before they actually do them and are obligated to publish their results no matter how unimpressive they are. That way there can be no cherry picking of results. Often papers are watered down and different (lesser) measures of outcomes are stressed because the main outcomes did not have the expected change. I think this change in tact should be made explicit.
The problem is most journals don't want to publish unimpressive results.

There are better truth finding methods out there, most famously formal logic, but everyday empirical methods and rational ones outside of science are underestimated in how effective they can be (they are responsible for the majority of technology).

Formal logic is impotent because it deals with truths - truths that must be 'proven' by science (or other methods) first. The problem is that science doesn't prove truths. It attempts to prove them, but it doesn't ever prove them without a doubt. For a working philosophy, see pancritical rationalism.

That said, there have been many triumphs of theoretical (hypothetical) science, so... "method", but there are lots of limitations in the way science is currently practised. The big ones for me are:

The topics which are researched - funding is complicated, but based on what is politically (due to outside funding) and socially (since individuals need to make a choice) acceptable. The funding for certain medical disorders for example is way out of wack with the expected benefits to patients.
Likewise, it is difficult to find funding for controversial studies.
Secondly, there are many biases to do with the communication of scientific knowledge.

Even on the low level, there are many flaws to do with publishing and the peer review system.
I don't have the time to go into detail, but here is a few of my bookmarks (actually, I lost all my old ones, so..)

http://www.the-scientist.com/templa...year=2010&page=36&month=8&o_url=2010/8/1/36/1 "I Hate Your Paper"
http://www.plosone.org/article/info:doi/10.1371/journal.pone.0010271 "Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data"

There are better truth finding methods out there, most famously formal logic, but everyday empirical methods and rational ones outside of science are underestimated in how effective they can be (they are responsible for the majority of technology).

Formal logic is impotent because it deals with truths - truths that must be 'proven' by science first. The problem is that science doesn't prove truths. It attempts to prove them, but it doesn't ever prove them without a doubt. For a working philosophy, see pancritical rationalism.

That said, there have been many triumphs of theoretical (hypothetical) science, so...
 
Last edited:

erm

Permabanned
Joined
Jun 19, 2007
Messages
1,652
MBTI Type
INFP
Enneagram
5
Formal logic is impotent because it deals with truths - truths that must be 'proven' by science first.

I'm agreeing with what you are saying except for this part.

First of all, no truth must be proven by science. Science is a very effective form of empiricism, so it's a wise choice, but it does not have the monopoly on finding any truth.

Not only that, formal logic, e.g. mathematics, doesn't seem to rely on empiricism at all (debatable). Whereas empiricism certainly has to rely on formal logic.
 

lunalum

Super Senior Member
Joined
Dec 20, 2008
Messages
2,706
MBTI Type
ZNTP
Enneagram
7w6
Instinctual Variant
sp/so
By which method should we test the scientific method? ;)

I didn't understand the article... I stopped at the part about the drugs demonstrating less effect in the later tests. If they are less effective, then they are less effective, and lost some of their power as drugs, have they not?

Unless the method is venturing outside the realm of facts and on to personal preferences/opinions, there shouldn't be much of a problem. If it is about facts, something worthwhile will eventually be found by observation and experimentations.
 

nolla

Senor Membrane
Joined
May 22, 2008
Messages
3,166
MBTI Type
INFP
I'm not really sure what the scientific method is exactly. Is the aim of the method to get to the truth? If it is, it should be redefined. I don't see any too big problems with it as long as we keep in mind that the truths are statistical and that the method works only inside the frame of reference it is meant for. What I see as a problem is that average people like myself have no idea what the definition of truth in science is. It seems that whenever they get enough creditability for a theory, they can sell it to the public too easily.
 

entropie

Permabanned
Joined
Apr 24, 2008
Messages
16,767
MBTI Type
entp
Enneagram
783
method ? That word feels strange, does it mean something naughty ?
 

Octarine

The Eighth Colour
Joined
Oct 14, 2007
Messages
1,351
MBTI Type
Aeon
Enneagram
10w
Instinctual Variant
so
Not only that, formal logic, e.g. mathematics, doesn't seem to rely on empiricism at all.

Except that it does. The choices by which a formal system is constructed/tested are based on intuition and empiricism (I guess as test of perceived usefulness). Otherwise you get an infinite number of formal systems - pretty useless if you are a mathematician.

Oh and empiricism only relies on formal logic in a limited sense. Actually, in many respects formal logic is obsolete these days due to concepts such as complexity.

method ? That word feels strange, does it mean something naughty ?

It would be more correct to say methods, rather than method singular.
 

erm

Permabanned
Joined
Jun 19, 2007
Messages
1,652
MBTI Type
INFP
Enneagram
5
Except that it does. The choices by which a formal system is constructed/tested are based on intuition and empiricism (I guess as test of perceived usefulness). Otherwise you get an infinite number of formal systems - pretty useless if you are a mathematician.

Oh and empiricism only relies on formal logic in a limited sense. Actually, in many respects formal logic is obsolete these days due to concepts such as complexity.

Well, why don't you explain how some simple formal logic, like 2 + 2 = 4, is based on empiricism?

Also, perhaps explain how formal logic has become obsolete by complexity? If you mean regular complexity instead of say, mathematical complexity, then all truth finding methods suffer greatly because of it.

Empiricism cannot fully function without formal logic. Try processing data without mathematics, for example.

If it turns out randomness is inherent, then empiricism will truly have a boon over rationalism and formal logic. Otherwise, formal logic is the only truth finding method that preserves truth as it functions (meaning the error margin does not increase with each step).
 

Octarine

The Eighth Colour
Joined
Oct 14, 2007
Messages
1,351
MBTI Type
Aeon
Enneagram
10w
Instinctual Variant
so
The discovery of addition was absolutely driven by empiricism.
 

erm

Permabanned
Joined
Jun 19, 2007
Messages
1,652
MBTI Type
INFP
Enneagram
5
Care to back that up?

I can back up formal logic not being based on, nor driven by empiricism simply by referring to how a logic puzzle is solved, an equation balanced or an incompleteness theorem discovered. Not through observation nor induction, but through deduction.
 

Lateralus

New member
Joined
May 18, 2007
Messages
6,262
MBTI Type
ENTJ
Enneagram
3w4
I've always thought people have been too confident in scientific studies of complex systems. I know that scientists do their best to isolate a single variable, but that's difficult to do with relatively simple systems (like some physics experiments). When you are conducting experiments on complex systems, like the human body, it is impossible.

I'm not saying that we need to toss science aside, but we need to reevaluate how much confidence give scientific studies.
 
Top