EUVIMEDEUVIMED entdecken
Randomisierte kontrollierte StudieOpen Access · CC BY 4.0

Testing for reviewer anchoring in peer review: A randomized controlled trial

Ryan Liu iD, Steven Jecmen, Vincent Conitzer, Fei Fang, Nihar B Shah iD

PLOS ONE · 2024

DOI: 10.1371/journal.pone.0301111 ↗

Kurz gefasst

Randomisierte kontrollierte Studie in der Zeitschrift PLOS ONE, 2024. Verfasst von Ryan Liu und 4 weiteren Autor:innen. Beteiligte Einrichtung: School of Computer Science, Carnegie Mellon University, Pittsburgh, Pennsylvania, United States of America. Hauptthemen laut MEDLINE: Peer Review, Research. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 59 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NSF, National Science Foundation, Division of Information and Intelligent Systems.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Peer review frequently follows a process where reviewers first provide initial reviews, authors respond to these reviews, then reviewers update their reviews based on the authors’ response. There is mixed evidence regarding whether this process is useful, including frequent anecdotal complaints that reviewers insufficiently update their scores. In this study, we aim to investigate whether reviewers anchor to their original scores when updating their reviews, which serves as a potential explanation for the lack of updates in reviewer scores.

Design

We design a novel randomized controlled trial to test if reviewers exhibit anchoring. In the experimental condition, participants initially see a flawed version of a paper that is corrected after they submit their initial review, while in the control condition, participants only see the correct version. We take various measures to ensure that in the absence of anchoring, reviewers in the experimental group should revise their scores to be identically distributed to the scores from the control group. Furthermore, we construct the reviewed paper to maximize the difference between the flawed and corrected versions, and employ deception to hide the true experiment purpose.

Results

Our randomized controlled trial consists of 108 researchers as participants. First, we find that our intervention was successful at creating a difference in perceived paper quality between the flawed and corrected versions: Using a permutation test with the Mann-Whitney U statistic, we find that the experimental group’s initial scores are lower than the control group’s scores in both the Evaluation category (Vargha-Delaney A = 0.64, p = 0.0096) and Overall score ( A = 0.59, p = 0.058). Next, we test for anchoring by comparing the experimental group’s revised scores with the control group’s scores. We find no significant evidence of anchoring in either the Overall ( A = 0.50, p = 0.61) or Evaluation category ( A = 0.49, p = 0.61). The Mann-Whitney U represents the number of individual pairwise comparisons across groups in which the value from the specified group is stochastically greater, while the Vargha-Delaney A is the normalized version in [0, 1].

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Peer Review, ResearchHumansPeer ReviewFemaleMale

Autor:innen und Institutionen

  • School of Computer Science, Carnegie Mellon University, Pittsburgh, Pennsylvania, United States of America.

ORCID-Profile: Ryan Liu, Nihar B Shah

Förderung

  • NSF
  • National Science Foundation
  • Division of Information and Intelligent Systems

Zitieren

APA

Liu, R., Jecmen, S., Conitzer, V., Fang, F., & Shah, N. B. (2024). Testing for reviewer anchoring in peer review: A randomized controlled trial. PLOS ONE. https://doi.org/10.1371/journal.pone.0301111

Vancouver

Liu R, Jecmen S, Conitzer V, Fang F, Shah NB. Testing for reviewer anchoring in peer review: A randomized controlled trial. PLOS ONE. 2024. doi:10.1371/journal.pone.0301111

BibTeX
@article{Liu2024,
  author = {Liu, Ryan and Jecmen, Steven and Conitzer, Vincent and Fang, Fei and Shah, Nihar B},
  title = {Testing for reviewer anchoring in peer review: A randomized controlled trial},
  journal = {PLOS ONE},
  year = {2024},
  doi = {10.1371/journal.pone.0301111}
}
Kostenlos starten →