With the current dramatic increases in both the number of prestigious scientific journals and the number of articles being submitted to those journals, editors must be on the lookout for new ways to combat peer reviewer fatigue. And these trends mean that it is becoming increasingly necessary for at least some of the tasks associated with the peer review process to be automated, just to reduce the burden on human editors and reviewers.
But as is the case with anything related to AI, there’s a fine line between where automation is helpful, versus where it is unhelpful at best and a hindrance at worst. The question then becomes: Which tasks in peer review can be streamlined using automation, and where must human help still be top priority?
Where Does Automation Help?
Sending Invitations
As peer review fatigue sets in, editors are being forced to send increasing numbers of invitations to review, because many of those invitations will, inevitably, be declined. But when those editors are constantly having to manually check to see whether more invitations are needed, it adds an even more time-consuming layer to the process.
This is where automation can come in: After a certain amount of time, invitations can be automatically declined (since at some point, it’s unlikely to receive a response). And if only a certain number of selected reviewers are invited, with the others on the list as backup (as is often the case), those “plan B” reviewers can be automatically invited as soon as the final open invitations are declined, whether by the reviewer or automatically. Once all potential options have been exhausted, an automatic popup can then indicate to the editor that more reviewer selections are needed. This saves the editor of having to tediously comb through queues to see which manuscripts need more reviewers and when.
Identifying Conflicts of Interest
Theoretically, authors should be disclosing conflicts of interest upon submission of an article to a scholarly journal. Realistically, though, this doesn’t always happen—and oftentimes, it’s just an honest mistake rather than an intentional omission. But when reviewers are busy judging the content of the article, it can be tough for them to even identify financial conflicts. This is where automation comes in; automated/AI tools can hunt for keywords/phrases suggesting the potential for undisclosed COI. They might also be able to scan previous works by an author group to detect patterns of repeated collaborations that could be red flags. Of course, anything that’s flagged by automation will subsequently need to be checked over by a human; still, though, it provides at least some help in pointing out problems that might otherwise have been missed entirely.
Weeding Out Candidates for Desk Rejection
If a paper does not comply with journal policy, and/or its subject matter is completely outside the journal’s scope, it’s most likely not worth sending for peer review at all. Often, editors will spend hours (or even days) just determining which manuscripts on their desks fall under this category, when automation can instead make that determination in a matter of minutes.
Although it might not be wise to totally rely on automation to identify keywords and phrases to select competent reviewers (more on this in the next section), this kind of method can be helpful in screening for those papers that aren’t good candidates for peer review at all. If, for example, a journal’s subject matter focuses on marine life, but automation determines through keywords screening that the paper is instead a human health study, sending it for review would be a waste of time for everyone involved—the authors, the reviewers, and the editor.
Of course, once automation does its job, it’s still wise for the editor to take a quick glance, just to ensure no mechanical errors were made and that something worth reviewing doesn’t slip through the cracks. But the amount of time spent doing this still decreases dramatically when automation is involved.
Where Do Humans Still Matter?
Selecting Subject Matter Experts as Potential Reviewers
It can be tempting, in the age of AI, to simply type subject matter keywords into some kind of AI tool, letting that tool find potential reviewers to invite. And these kinds of tools definitely can be helpful in at least bringing up helpful suggestions.
Still, it will always be necessary for a human to check that tool’s work and do their own research to determine whether those potential reviewers are a good fit for a scientific article. Letting automation completely take over the task will inevitably lead to subject matter mismatches—it is simply impossible for keywords and phrases to always churn out appropriate candidates. The result will just be more invitations that are either declined (with reasons such as, “Sorry, this is out of my area of expertise”) or not responded to at all—and that creates more work for the editor, not less.
Keeping Research Data Confidential
Picture the following scenario (which, while hypothetical, is becoming more common): In an effort to streamline the process of writing reviews for dozens of papers by a certain deadline, a reviewer enlists the help of an automated AI service to scan the paper and write an outline, based on similar research that service might find online. The reviewer does plan to evaluate the paper too, and will use this automation-generated outline only as a launch point. This is permissible—right?
…Unfortunately, it’s not that simple. Even if the reviewer does plan only to use automation/AI as a place to start, by feeding the article into that automated service, they are potentially exposing the unpublished research to the entire internet. This could result in confidential (or perhaps even inaccurate) scientific data being spread prematurely across multiple online platforms, which could create significant ethical issues months or even years down the road.
Committing Decisions
After the peer review process is complete, it has traditionally been the job of the editor (and/or a member of the editorial staff) to manually send the letter to the author with the review comments, along with the final decision. In recent years, the idea of using an automated system to complete this task has been discussed increasingly often. However, this comes with numerous potential problems. First, it does not allow the editor to customize the decision letter with comments about why that particular decision was made, which can be extremely important. Moreover, automated systems don’t always catch issues of reviewer confidentiality (since, generally speaking, reviewer identities are supposed to be hidden from authors). A reviewer might, for instance, upload a review as an attachment, and that attachment might include his/her name in the metadata of the Word comments file. While a human screener would (at least theoretically) catch this and strip it out, an automated system might miss it, thus compromising the confidentiality—and, thus, the overall ethics—of the process. So, it likely will always be necessary for a human set of eyes to at least skim decision letters before review comments are returned to authors.
Conclusion
Automation can be of great benefit in aiding the peer review process—but it will never be a replacement for human peer review entirely. And unfortunately, there are already entirely too many bad actors attempting to get away with using automation to do the bulk of peer review work for them. It will become increasingly more important in the coming years for journal editorial boards to identify when this occurs and create safeguards to prevent it from happening, so as to keep the spirit of peer review what it was meant to be: a catalyst to ensure only the best research makes it into scholarly literature.
By Anne Brenner
