Peer-Review Primer

This resource provides an introduction to and overview of peer-review, the basics of how peer-review and decision making is organized at journals, and the nuances of different peer-review models.

What is Peer-Review?

Peer-review is a process used by scholarly journals and edited books to assess the overall quality and importance of the work submitted, and select appropriate material for publication. Some scholarly conferences with an open call for contributions also assess submitted work using peer-review. Traditionally, the goal of the peer-review process is to evaluate a paper in regards to its scientific merit, significance, novelty, and presentation. Peer-review is thus performed by qualified experts that are actively involved in scholarly or scientific research in the same field.

Typically, peer-review is conducted anonymously, meaning that the referees’ identities are not revealed to the authors (single blind peer-review). Some journals use a double blind peer-review model, where additionally the authors’ identities are masked in the paper prior to sending it to the reviewers.

The anonymous nature of peer-review allows for referees to be critical of papers they deem insufficient compared to commonly accepted scientific standards. However, it can also introduce a bias, where referees competing in the same field are dismissing or unnecessarily delaying the publication of scientifically sound papers, only to have the chance to publish similar research results first.

Double blind peer-review aims at removing referee bias that discriminates papers written by certain types of authors (e.g. making distinctions by race, religion, gender, etc.), while encouraging a clear focus on the content and scientific merit of the paper.

Most journals’ peer-review is done before publication of an article and is single blinded. However, due to growing critique of the traditional peer-review process, in recent years, new innovative forms of peer-review have emerged, such as open peer-review, collaborative peer-review and post-publication peer-review:

• Open peer-review models make referee reports available electronically to the readers alongside the published paper. Some open peer-review models also require referee identities to be revealed to authors or the public, which is usually done once the peer-review process has been completed.
• Collaborative peer-review designates a model where referees, authors and editors exchange, discuss and debate the paper in an online forum, before the editors make a decision and eventually ask authors to revise the paper.
• Under the post-publication peer-review (PPPR) model a paper is immediately published, before referees or the scientific community at large are asked to publicly evaluate and comment on the paper.

Basics of Peer-Review and Decision Making for Journals

The journal staff or editors will select qualified experts to be invited for peer-review. Referees are usually selected from the journal‘s editorial board members, from the database of past and volunteer reviewers, by looking up authors of papers cited in the references section, or by searching for authors of related papers in indexing and abstracting databases. Some journals also ask authors to provide the names of a few potential referees, or to indicate persons they want to exclude from peer-reviewing their work. Typically, two referee reports and sometimes more are collected per paper, before an academic editor of the journal will render a decision. Editors may seek additional opinions on a paper, if the first few referees are in strong disagreement, or if one of the review reports is found to be strongly biased or otherwise unusable (e.g. a very short report).

Once enough referee reports have been collected, one or more academic editors of the journal will make a decision regarding the submitted paper. For most journals the decision is solely the editor’s responsibility: the comments by referees are complementary advice and the editor may choose to overrule referees. Other journals may render decisions on papers during a virtual or physical editorial board meeting, or by correspondence. Typically, the journal’s editors will provide a short rationale for the decision when passing the referee reports on to the authors.

Most journals distinguish the following decisions: desk rejection, rejection, revisions with or without additional rounds of peer-review, and acceptance:

• A desk rejection consists in rejecting a paper just after the submission and before it is sent out for peer-review. Desk rejections filter out papers that do not fit the journal, are poorly presented, have critical language issues, are obviously not up to scientific standards, or are unnecessarily controversial. Where papers are desk rejected due to poor presentation or language issues, editors may choose to invite the authors to resubmit the paper to the same journal.
• A rejection is pronounced after one or more rounds of peer-review. Referees may agree that a paper is not of a publishable standard during the first round of review. Often a rejection is given after the first or second round of revisions, in case authors failed to address the concerns raised by the referees and/or the editors. It happens sometimes that authors disagree with referees, or think that the referees’ demands are excessive. In such cases, most journals have a policy that allows authors to appeal the rejection. Often with such policies, an appropriate editorial board member will be selected to either deliver an additional review report, or to render a final, non-contestable decision based on the previously collected peer-review reports. If you want to appeal a rejection decision, study the journal’s editorial policies carefully to properly build a case.
• Most papers will receive a “revisions” decision. Some journals may differentiate between several levels of revisions, such as minor revisions and major revisions. Minor revisions can be interpreted as a conditional acceptance. Revisions will typically be focused at changing a few presentational aspects, detailing a procedure, rewriting a few paragraphs, Such changes can be appraised by the journal’s editors without having to send the revised paper back to the referees. Major revisions may include rewriting an entire section of the paper, conducting additional experiments, re-analyzing the experimental results, etc. These entail substantial changes to the paper that can take several weeks, if not months, to complete. As such, these revisions have to be reevaluated by the referees in a second review round. The outcome of a second review round can be anything from a rejection, the requirement for additional revisions, to an acceptance.
• Papers may eventually be accepted after one or more rounds of peer-review. Depending on the journal, the acceptance can be decided by the manuscript handling editor, the chief editor, or by an editorial committee or editorial board meeting.

Peer-Review Problems

Peer-review is an essential system in scholarly communications. Yet, it has some shortcomings:

• Peers naturally compete for impact and fame. Under the traditional, anonymous peer-review model, this can cause referees to intentionally hold back or dismiss papers that report significant new findings, only to have the chance to publish similar results first. The process lacks a mechanism, other than the editor’s insistence, to pressure referees to be impartial. Similarly, we lack mechanisms to reward referees for good behavior.
• Further, referees may have a bias towards favoring papers from well-known, established authors, and discriminate papers by gender, race, religion, lack of status, their controversial nature and other characteristics of the authors.
• Some journals ask authors to provide the names of a few potential referees when they submit their paper. This has led to abuse in the past, where authors have created e-mail addresses and submitted fake referee profiles to the publisher in order to later review their own papers. RetractionWatch, a blog that documents retractions of scholarly journals and scientific misconduct, lists more than 100 retractions for “self peer-review”. Even well-established publishing houses and journals have been hit by this form of peer-review scam.
• Last but not least, the pressure on peer-reviewers has generally increased over the past years: the global research output has almost doubled during the last ten years, while the pool of qualified referees has been growing much slower.

As a consequence, some argue that the peer-review process is flawed and unnecessarily slows down research. Some journals have thus started open peer-review practices (see next section for types of peer-review). Open peer-review represents a strong mind-shift, and thus many journals and editors have some reservations in adopting this model in fear of scaring away referees in a context where it is already difficult to find qualified referees.

Types of Peer-Review

Traditional Peer-Review (Single Blind Peer-Review)

Under the traditional, single blind peer-review model, the referees’ identity is not revealed to authors. However, referees have access to the names of the authors. Traditional, single blind peer-review is conducted before publication of an article: the process leads to the actual acceptance or rejection decision.

Double Blind Peer-Review

Like for traditional, single blind peer-review, the identities of referees are not revealed to authors. Additionally, with double blind peer-review, papers are obfuscated before being sent out for review: authors’ names are deleted from the manuscript. Where authors self-cite their previous papers, language may be altered to mask the identity of authors. Journals that operate double blind peer-review will typically ask authors to submit two versions of their manuscript: the full manuscript, and the manuscript where authors’ identities have been obfuscated.

Double blind peer-review aims at removing a referee bias: referees may not discriminate papers on the basis of race, religion, gender, status of the author, previous controversial papers by the author, etc. Instead, double blind peer-review promotes a focus on content. Double blind peer-review can thus be an interesting model for younger researchers or members of minority groups. Unfortunately, in practice, it is often difficult to fully mask the identities of the authors and, thus, referees can often guess who the authors are.

Collaborative Peer-Review

With collaborative peer-review, authors, referees and editors will discuss and debate the paper in an online forum, before editors render a decision on the paper. Collaborative peer-review is an iterative process between authors and referees, with much shorter and more feedback loops compared to traditional peer-review. Collaborative peer-review has a strong focus on improving papers in a short period of time. Swiss-based open access publisher Frontiers is best known for applying a two phase review system, where the second phase is a collaborative one.

Open Peer-Review

With open peer-review, the records of the peer-review process are published together with the paper, sometimes with the names of the referees. This move helps journals to transparently document the peer-review process for each published paper, and—where referee names are published—forces referees to be considerate. Reports for papers that were rejected are typically not published.

Post-Publication Peer-Review (PPPR)

With post-publication peer-review, papers are published first and then commented on by invited referees or the scientific community at large. Examples of journals operating the PPPR model include the open access journals F1000 Research and ScienceOpen Research. Post-publication peer-review is usually done publicly (thus PPPR is also a form of open peer-review). Any paper submitted to these journals will be published first, and referees will then be invited to publicly review the paper. Authors are usually given the chance to revise their paper after a few months, based on the feedback from referees.

However, post-publication peer-review works with any published paper: there are several platforms, such as PubPeer or PubMed Commons, which allow anyone to post comments and discuss published papers. PubPeer’s mission for example is to “help improve the quality of scientific research by enabling innovative approaches for community interaction”. It has attracted more than 35,000, mostly anonymous, comments to date and has become an important vehicle for making allegations of scientific misconduct, according to a Nature News piece published in August 2015.