Peer review: A Flawed Process?

[Doctor Cringelord]

Peer review: a flawed process at the heart of science and journals

I don't know. Mixed thoughts on the topic. On one hand it's important that scientific claims be backed by research and reviewed by other professionals in an impartial manner. On the other hand, the idea of placing faith in the subjective opinions of others or trusting others' words on the merits of scientific findings smacks of orthodoxy, and when science becomes orthodoxy, it's no longer science at that point.

 

[Coriolis]

Peer review: a flawed process at the heart of science and journals

I don't know. Mixed thoughts on the topic. On one hand it's important that scientific claims be backed by research and reviewed by other professionals in an impartial manner. On the other hand, the idea of placing faith in the subjective opinions of others smacks of orthodoxy, and when science becomes orthodoxy, it's no longer science at that point.

1. If reviewers are basing their reviews on subjective opinions, they are not doing their job right.

2. What is the opposite of a flawed process? A perfect process? No human endeavor is perfect, so that is the bottom line right there.

3. Why is this thread in the politics subforum instead of science?

 

[Maou]

I've been saying for years that the system has been infected with political interests, and agendas affecting what is considered "legit" science. Just because something is a majority, doesn't mean it is right either. Scientists are in no way any more objective and rational than any other group of humans. Why are they held on such a pedestal of honest integrity? They sure as fuck would take money and publish a bunk study, and get their buddies to agree with it for money as well. They can also be strong armed into publishing shit they know is bunked, by being threatened with defunding or losing credentials. The people doing the pressuring and bribing know fuck well that no one can stop them.

 

[rav3n]

This paper was published in 2006. Does anyone know if anything's changed since?

 

[Doctor Cringelord]

Oops, did not mean to post this in the politics sub forum. Apologies

 

[Jaq]

1. If reviewers are basing their reviews on subjective opinions, they are not doing their job right.

2. What is the opposite of a flawed process? A perfect process? No human endeavor is perfect, so that is the bottom line right there.

Agreed.

No human work is perfect as humans are inherently flawed, but duplication for experiments, for example is important. I'm not a scientist by any stretch of the imagination, but I would say that it's better than scientists making random claims all willy-nilly and such without anyone able to critique it.

 

[Doctor Cringelord]

Agreed.

No human work is perfect as humans are inherently flawed, but duplication for experiments, for example is important. I'm not a scientist by any stretch of the imagination, but I would say that it's better than scientists making random claims all willy-nilly and such without anyone able to critique it.

Ironically, one of the criticisms people have made about the peer review system is that it doesn't always properly vet papers and their findings, that poorly researched papers make it to publication based on the whims of those reviewing them.

Also, peer review doesn't often involve duplication to confirm experiment findings in the first place. It's a process often relying on unpaid review work by professionals who are already busy with their own workloads--as such, it can be a drawn out process causing publication of potentially important findings to be delayed for years. Not to mention in some cases, those tasked with peer review are often in direct competition with the authors of papers being submitted. Can't imagine any bias or conflict of interest resulting in that type of situation.

 

[Doctor Cringelord]

Let's stop pretending peer review works - Vox

 

[Jaq]

Ironically, one of the criticisms people have made about the peer review system is that it doesn't always properly vet papers and their findings, that poorly researched papers make it to publication based on the whims of those reviewing them.

Also, peer review doesn't often involve duplication to confirm experiment findings in the first place.

Ah, thanks for the response.

 

[Doctor Cringelord]

I don't advocate doing away with the peer review system, but I'd make it more impartial. We could start by making the names of the authors and institutions responsible for various studies unknown to those reviewing the studies and papers. They should be reviewing the findings with minimal bias, and that would be a great start.

 

[Coriolis]

Ironically, one of the criticisms people have made about the peer review system is that it doesn't always properly vet papers and their findings, that poorly researched papers make it to publication based on the whims of those reviewing them.

Also, peer review doesn't often involve duplication to confirm experiment findings in the first place.

Peer reviewers typically do not have the resources (time, money, specific expertise) to duplicate the results of even a fraction of the papers they review. A responsible publication will be able to provide adequate evidence of reproducibility by the authors, and consistency (or explained inconsistencies) with previous work. It usually falls to other research groups to attempt to duplicate the work, work they may find out about mainly through publication. So, publishing something is often a necessary step to having it duplicated - or refuted - by others.

If the complaint is about undue influence of politics on science, this is hardly an indictment of scientific method itself. It is a complaint many scientist also share, and is rooted in the fact that money drives research, just like anything else. If scientists are to be free to judge publications exclusively on their scientific merits, they must be insulated from such financial constraints, a pipe dream in today's world.

I don't advocate doing away with the peer review system, but I'd make it more impartial. We could start by making the names of the authors and institutions responsible for various studies unknown to those reviewing the studies and papers. They should be reviewing the findings with minimal bias, and that would be a great start.

This will be near impossible given the level of specialization involved in much of contemporary research, and the fact that citing prior work (including one's own) is a key component of responsible research and the reporting thereof. Papers are not job applications, and are best reviewed in the context of their field.

 

[Tellenbach]

I'd recommend The Hockey Stick Illusion for those who want to delve more into the corruption of science by the gatekeepers of various scientific journals. It's more than just the lack of time and resources to duplicate work; some of the senior editors of prestigious journals refuse to publish papers that stray from the dominant, consensus opinion. This becomes a problem because science is supposed to be evidence driven, not popularity driven.

If someone takes the time to attempt duplication of another team's work, they're going to be ignored if it's a politically incorrect position.

 

[Pionart]

One reason that I haven't gone into academia is that by doing so I would have to censor my opinions and produce work I'm not happy with in order to have it approved.

 

[Vendrah]

I did have years ago somewhat of a class about scientific approach.
Nothing deep, just quickly stuff and I forgot a good portion of it.

The big thing about classic physics and Newtonian physics is that a lot of the experiments were cheaper, and, different then it is from humanities and bios, they are supposed to work a 100% of the time, and if they dont, it is basically the experimenter's fault or fault of instruments. So, basically, these "primordial" scientific experiments could be replicated hundreds of times. This is the "hardcore" part of science, that is indeed rigorous, as mostly physics and chemistry are. And that is a quick and simple yet superficial way of explaining the "scientific method" and how it should be: Different and independent sources replicating the experiment.

However, with time experiments got expensive and expensive, and where some experiment were just basic mixing chemical stuff and observing colours, particle experiments went by and these can only be done like... One time in a decade in only one place. And, of course, later bios and then humanities followed by and these cant really keep up with the rigorous standards that earlier chemistry and physic could come by. While if you throw a stone in your open window to a clear place it will fall a 100% of the time, if you go for even the 'best' versions of MBTI tests the test-retest rate will be only 60%. A lot of medical experiments are on the middle zone: They get a better repeatability than MBTI but less than Newtonian physics.

With that in account, I do agree that the peer review as described in the link by the OP isnt rigorous as it should, but I think people might be confusing a few things here. Im sorry if it sounds a little arrogant, but physics and chemistry, at least in terms of experiments, are above all other sciences. Not necessarily because people on other sciences are incompetent, but due to repeatability. So, in one extreme, comparing Newtonian physics with MBTI, taking a few failure experiments and procedures from the medical department does not cancel the whole science and make it less scientific and "yaaayyy, science isnt like that, lets pay attention to stupid conspiracy things and pretend we are rational". Things arent black and white like that, it pretty much depends in which part of science you are talking to. If it is MBTI that barely or perhaps not even make it, or Newtonian experiments that have been replicated literally thousands of times (correction: Actually, Newtonian physics doesnt work properly on particle physics and objects close to speed of light, so it doesnt work a 100% of the times, but it will in normal size and speeds that are slow when compared to the speed of light).

Repeatability is indeed the most appropriate way and it would be indeed the best peer-review; But it gets too costy sometimes. Because of that, liking or not, like many things in life, honesty, sincerity, is something of vital importance. At the same time that a country, regardless of having a small state or a big state, is very prone to succumb to internal dishonesty, soft sciences are very prone to dishonesty as well. If people want to fool for money, for prestige, whatever... It will indeed ruin things. I dont think there is really any truly objective solution to this, but there is integrity. This happens somewhat to data analysis as well.

 

[Coriolis]

Repeatability is indeed the most appropriate way and it would be indeed the best peer-review; But it gets too costy sometimes. Because of that, liking or not, like many things in life, honesty, sincerity, is something of vital importance. At the same time that a country, regardless of having a small state or a big state, is very prone to succumb to internal dishonesty, soft sciences are very prone to dishonesty as well. If people want to fool for money, for prestige, whatever... It will indeed ruin things. I dont think there is really any truly objective solution to this, but there is integrity. This happens somewhat to data analysis as well.

Repeatibility is essential in scientific inquiry - so essential, in fact, that it cannot and generally is not left to the peer review process. Any kid doing a science fair project knows to repeat the experiment several times. This is the most basic demonstration of repeatibility. How many times must it be repeated? Until the conclusions are statistically significant, meaning until the data display minimal scatter and point to one result with reasonable confidence. (Yes, "significant", "minimal", and "reasonable" are qualitative terms, and must be quantified based on the nature of the experiment and the data collected.)

Professional researchers should be and are held to the same standard. A research group that cannot demonstrate repeatability in their own work is not ready to publish, and should dig deeper into their experiment and theory to figure out what is going on. Failure to do this is grounds for downgrading a manuscript, or rejecting it altogether. Typically the review process will provide detailed feedback to the authors, to help them address any deficiencies and improve their work for publication later on.

 

[Coriolis]

Getting published by a "true" journal is really hard because editors will ensure the repeatibility Coriolis is talking about. And even after that, papers can still get pulled back due to inconsistencies, mistakes, or the very same peer review, but not because of someone from above who payed/bribes.

Not only that, but reputable journals will print corrections or addenda if something was seriously amiss.

I meant to add somewhere that I have been on both sides of the peer review process. I have submitted publications for peer review and had them published, and I have served as a peer reviewer. Not that that makes me an authority, and practices might be a bit different in different fields, but it does give me direct insight into the process. The biggest problem in my field is that too many papers get published which make only incremental progress, in other words, not what I could consider sufficient to warrant publication. The work is scientifically sound and well documented, just doesn't make that much of a contribution to the field beyond what has already been published. I generally recommend against publishing such papers, and provide the authors suggestions on how to expand their work to have a greater impact. Editors, however, usually do not take my recommendation.