• You are currently viewing our forum as a guest, which gives you limited access to view most discussions and access our other features. By joining our free community, you will have access to additional post topics, communicate privately with other members (PM), view blogs, respond to polls, upload content, and access many other special features. Registration is fast, simple and absolutely free, so please join our community today! Just click here to register. You should turn your Ad Blocker off for this site or certain features may not work properly. If you have any problems with the registration process or your account login, please contact us by clicking here.

Peer review: A Flawed Process?

Doctor Anaximander

Well-known member
Joined
Aug 27, 2013
Messages
19,167
Enneagram
5
Instinctual Variant
sx/sp
Peer review: a flawed process at the heart of science and journals

I don't know. Mixed thoughts on the topic. On one hand it's important that scientific claims be backed by research and reviewed by other professionals in an impartial manner. On the other hand, the idea of placing faith in the subjective opinions of others or trusting others' words on the merits of scientific findings smacks of orthodoxy, and when science becomes orthodoxy, it's no longer science at that point.
 

Cor Luctis

Between the Shadows
Staff member
Joined
Apr 18, 2010
Messages
26,583
MBTI Type
INTJ
Enneagram
5w6
Instinctual Variant
sp/sx
Peer review: a flawed process at the heart of science and journals

I don't know. Mixed thoughts on the topic. On one hand it's important that scientific claims be backed by research and reviewed by other professionals in an impartial manner. On the other hand, the idea of placing faith in the subjective opinions of others smacks of orthodoxy, and when science becomes orthodoxy, it's no longer science at that point.
1. If reviewers are basing their reviews on subjective opinions, they are not doing their job right.

2. What is the opposite of a flawed process? A perfect process? No human endeavor is perfect, so that is the bottom line right there.

3. Why is this thread in the politics subforum instead of science?
 

тень

Eclipsing
Joined
Jun 20, 2018
Messages
5,844
MBTI Type
TiNi
Enneagram
649
Instinctual Variant
sp
I've been saying for years that the system has been infected with political interests, and agendas affecting what is considered "legit" science. Just because something is a majority, doesn't mean it is right either. Scientists are in no way any more objective and rational than any other group of humans. Why are they held on such a pedestal of honest integrity? They sure as fuck would take money and publish a bunk study, and get their buddies to agree with it for money as well. They can also be strong armed into publishing shit they know is bunked, by being threatened with defunding or losing credentials. The people doing the pressuring and bribing know fuck well that no one can stop them.
 

Methylene

Now with less salt.
Joined
Oct 10, 2016
Messages
1,821
MBTI Type
LGBT
Enneagram
639
Instinctual Variant
so/sp
I've been saying for years that the system has been infected with political interests, and agendas affecting what is considered "legit" science. Just because something is a majority, doesn't mean it is right either. Scientists are in no way any more objective and rational than any other group of humans. Why are they held on such a pedestal of honest integrity? They sure as fuck would take money and publish a bunk study, and get their buddies to agree with it for money as well. They can also be strong armed into publishing shit they know is bunked, by being threatened with defunding or losing credentials. The people doing the pressuring and bribing know fuck well that no one can stop them.

Please bribe me, I beg you.

I'll post something serious later, since I'm studying to work as a researcher and I know many who already do.
 

Jaq

Remember, Humanity.
Joined
Apr 14, 2011
Messages
3,040
MBTI Type
ENTP
Enneagram
379
Instinctual Variant
sp/sx
1. If reviewers are basing their reviews on subjective opinions, they are not doing their job right.

2. What is the opposite of a flawed process? A perfect process? No human endeavor is perfect, so that is the bottom line right there.

Agreed.

No human work is perfect as humans are inherently flawed, but duplication for experiments, for example is important. I'm not a scientist by any stretch of the imagination, but I would say that it's better than scientists making random claims all willy-nilly and such without anyone able to critique it.
 

Doctor Anaximander

Well-known member
Joined
Aug 27, 2013
Messages
19,167
Enneagram
5
Instinctual Variant
sx/sp
Agreed.

No human work is perfect as humans are inherently flawed, but duplication for experiments, for example is important. I'm not a scientist by any stretch of the imagination, but I would say that it's better than scientists making random claims all willy-nilly and such without anyone able to critique it.

Ironically, one of the criticisms people have made about the peer review system is that it doesn't always properly vet papers and their findings, that poorly researched papers make it to publication based on the whims of those reviewing them.

Also, peer review doesn't often involve duplication to confirm experiment findings in the first place. It's a process often relying on unpaid review work by professionals who are already busy with their own workloads--as such, it can be a drawn out process causing publication of potentially important findings to be delayed for years. Not to mention in some cases, those tasked with peer review are often in direct competition with the authors of papers being submitted. Can't imagine any bias or conflict of interest resulting in that type of situation.
 

Jaq

Remember, Humanity.
Joined
Apr 14, 2011
Messages
3,040
MBTI Type
ENTP
Enneagram
379
Instinctual Variant
sp/sx
Ironically, one of the criticisms people have made about the peer review system is that it doesn't always properly vet papers and their findings, that poorly researched papers make it to publication based on the whims of those reviewing them.

Also, peer review doesn't often involve duplication to confirm experiment findings in the first place.

Ah, thanks for the response.
 

Doctor Anaximander

Well-known member
Joined
Aug 27, 2013
Messages
19,167
Enneagram
5
Instinctual Variant
sx/sp
I don't advocate doing away with the peer review system, but I'd make it more impartial. We could start by making the names of the authors and institutions responsible for various studies unknown to those reviewing the studies and papers. They should be reviewing the findings with minimal bias, and that would be a great start.
 

Cor Luctis

Between the Shadows
Staff member
Joined
Apr 18, 2010
Messages
26,583
MBTI Type
INTJ
Enneagram
5w6
Instinctual Variant
sp/sx
Ironically, one of the criticisms people have made about the peer review system is that it doesn't always properly vet papers and their findings, that poorly researched papers make it to publication based on the whims of those reviewing them.

Also, peer review doesn't often involve duplication to confirm experiment findings in the first place.
Peer reviewers typically do not have the resources (time, money, specific expertise) to duplicate the results of even a fraction of the papers they review. A responsible publication will be able to provide adequate evidence of reproducibility by the authors, and consistency (or explained inconsistencies) with previous work. It usually falls to other research groups to attempt to duplicate the work, work they may find out about mainly through publication. So, publishing something is often a necessary step to having it duplicated - or refuted - by others.

If the complaint is about undue influence of politics on science, this is hardly an indictment of scientific method itself. It is a complaint many scientist also share, and is rooted in the fact that money drives research, just like anything else. If scientists are to be free to judge publications exclusively on their scientific merits, they must be insulated from such financial constraints, a pipe dream in today's world.

I don't advocate doing away with the peer review system, but I'd make it more impartial. We could start by making the names of the authors and institutions responsible for various studies unknown to those reviewing the studies and papers. They should be reviewing the findings with minimal bias, and that would be a great start.
This will be near impossible given the level of specialization involved in much of contemporary research, and the fact that citing prior work (including one's own) is a key component of responsible research and the reporting thereof. Papers are not job applications, and are best reviewed in the context of their field.
 

Tellenbach

in dreamland
Joined
Oct 27, 2013
Messages
6,053
MBTI Type
ISTJ
Enneagram
6w5
I'd recommend The Hockey Stick Illusion for those who want to delve more into the corruption of science by the gatekeepers of various scientific journals. It's more than just the lack of time and resources to duplicate work; some of the senior editors of prestigious journals refuse to publish papers that stray from the dominant, consensus opinion. This becomes a problem because science is supposed to be evidence driven, not popularity driven.

If someone takes the time to attempt duplication of another team's work, they're going to be ignored if it's a politically incorrect position.
 

Legion

complete
Joined
Sep 17, 2014
Messages
3,420
MBTI Type
INF
Enneagram
945
One reason that I haven't gone into academia is that by doing so I would have to censor my opinions and produce work I'm not happy with in order to have it approved.
 

Vendrah

New forum night mode looks cool!
Joined
Mar 26, 2017
Messages
1,744
MBTI Type
NP
Enneagram
952
I did have years ago somewhat of a class about scientific approach.
Nothing deep, just quickly stuff and I forgot a good portion of it.

The big thing about classic physics and Newtonian physics is that a lot of the experiments were cheaper, and, different then it is from humanities and bios, they are supposed to work a 100% of the time, and if they dont, it is basically the experimenter's fault or fault of instruments. So, basically, these "primordial" scientific experiments could be replicated hundreds of times. This is the "hardcore" part of science, that is indeed rigorous, as mostly physics and chemistry are. And that is a quick and simple yet superficial way of explaining the "scientific method" and how it should be: Different and independent sources replicating the experiment.

However, with time experiments got expensive and expensive, and where some experiment were just basic mixing chemical stuff and observing colours, particle experiments went by and these can only be done like... One time in a decade in only one place. And, of course, later bios and then humanities followed by and these cant really keep up with the rigorous standards that earlier chemistry and physic could come by. While if you throw a stone in your open window to a clear place it will fall a 100% of the time, if you go for even the 'best' versions of MBTI tests the test-retest rate will be only 60%. A lot of medical experiments are on the middle zone: They get a better repeatability than MBTI but less than Newtonian physics.

With that in account, I do agree that the peer review as described in the link by the OP isnt rigorous as it should, but I think people might be confusing a few things here. Im sorry if it sounds a little arrogant, but physics and chemistry, at least in terms of experiments, are above all other sciences. Not necessarily because people on other sciences are incompetent, but due to repeatability. So, in one extreme, comparing Newtonian physics with MBTI, taking a few failure experiments and procedures from the medical department does not cancel the whole science and make it less scientific and "yaaayyy, science isnt like that, lets pay attention to stupid conspiracy things and pretend we are rational". Things arent black and white like that, it pretty much depends in which part of science you are talking to. If it is MBTI that barely or perhaps not even make it, or Newtonian experiments that have been replicated literally thousands of times (correction: Actually, Newtonian physics doesnt work properly on particle physics and objects close to speed of light, so it doesnt work a 100% of the times, but it will in normal size and speeds that are slow when compared to the speed of light).

Repeatability is indeed the most appropriate way and it would be indeed the best peer-review; But it gets too costy sometimes. Because of that, liking or not, like many things in life, honesty, sincerity, is something of vital importance. At the same time that a country, regardless of having a small state or a big state, is very prone to succumb to internal dishonesty, soft sciences are very prone to dishonesty as well. If people want to fool for money, for prestige, whatever... It will indeed ruin things. I dont think there is really any truly objective solution to this, but there is integrity. This happens somewhat to data analysis as well.
 

Cor Luctis

Between the Shadows
Staff member
Joined
Apr 18, 2010
Messages
26,583
MBTI Type
INTJ
Enneagram
5w6
Instinctual Variant
sp/sx
Repeatability is indeed the most appropriate way and it would be indeed the best peer-review; But it gets too costy sometimes. Because of that, liking or not, like many things in life, honesty, sincerity, is something of vital importance. At the same time that a country, regardless of having a small state or a big state, is very prone to succumb to internal dishonesty, soft sciences are very prone to dishonesty as well. If people want to fool for money, for prestige, whatever... It will indeed ruin things. I dont think there is really any truly objective solution to this, but there is integrity. This happens somewhat to data analysis as well.
Repeatibility is essential in scientific inquiry - so essential, in fact, that it cannot and generally is not left to the peer review process. Any kid doing a science fair project knows to repeat the experiment several times. This is the most basic demonstration of repeatibility. How many times must it be repeated? Until the conclusions are statistically significant, meaning until the data display minimal scatter and point to one result with reasonable confidence. (Yes, "significant", "minimal", and "reasonable" are qualitative terms, and must be quantified based on the nature of the experiment and the data collected.)

Professional researchers should be and are held to the same standard. A research group that cannot demonstrate repeatability in their own work is not ready to publish, and should dig deeper into their experiment and theory to figure out what is going on. Failure to do this is grounds for downgrading a manuscript, or rejecting it altogether. Typically the review process will provide detailed feedback to the authors, to help them address any deficiencies and improve their work for publication later on.
 

Methylene

Now with less salt.
Joined
Oct 10, 2016
Messages
1,821
MBTI Type
LGBT
Enneagram
639
Instinctual Variant
so/sp
Repeatibility is essential in scientific inquiry - so essential, in fact, that it cannot and generally is not left to the peer review process. Any kid doing a science fair project knows to repeat the experiment several times. This is the most basic demonstration of repeatibility. How many times must it be repeated? Until the conclusions are statistically significant, meaning until the data display minimal scatter and point to one result with reasonable confidence. (Yes, "significant", "minimal", and "reasonable" are qualitative terms, and must be quantified based on the nature of the experiment and the data collected.)

Professional researchers should be and are held to the same standard. A research group that cannot demonstrate repeatability in their own work is not ready to publish, and should dig deeper into their experiment and theory to figure out what is going on. Failure to do this is grounds for downgrading a manuscript, or rejecting it altogether. Typically the review process will provide detailed feedback to the authors, to help them address any deficiencies and improve their work for publication later on.

*I'll add something. I'll expand about the whole subject whenever I'll have the time.

There exist so called predatory journals that you can essentially pay to get published. But people working in the sector know what they are and straight discard articles from them. No exceptions. It's fun to see people who know nothing about science often quote those articles, though (ie: people against vaccines).
There are a few other useful criteria (such as avoiding articles written by only one person, general scores, etc).

Getting published by a "true" journal is really hard because editors will ensure the repeatibility Coriolis is talking about. And even after that, papers can still get pulled back due to inconsistencies, mistakes, or the very same peer review, but not because of someone from above who payed/bribes.
 

Cor Luctis

Between the Shadows
Staff member
Joined
Apr 18, 2010
Messages
26,583
MBTI Type
INTJ
Enneagram
5w6
Instinctual Variant
sp/sx
Getting published by a "true" journal is really hard because editors will ensure the repeatibility Coriolis is talking about. And even after that, papers can still get pulled back due to inconsistencies, mistakes, or the very same peer review, but not because of someone from above who payed/bribes.
Not only that, but reputable journals will print corrections or addenda if something was seriously amiss.

I meant to add somewhere that I have been on both sides of the peer review process. I have submitted publications for peer review and had them published, and I have served as a peer reviewer. Not that that makes me an authority, and practices might be a bit different in different fields, but it does give me direct insight into the process. The biggest problem in my field is that too many papers get published which make only incremental progress, in other words, not what I could consider sufficient to warrant publication. The work is scientifically sound and well documented, just doesn't make that much of a contribution to the field beyond what has already been published. I generally recommend against publishing such papers, and provide the authors suggestions on how to expand their work to have a greater impact. Editors, however, usually do not take my recommendation.
 
Top