EUVIMEDEUVIMED entdecken
StudienprotokollOpen Access · CC BY 4.0

Registered report protocol: Factors associated with inter-rater agreement in grant peer review

Jan-Ole Hesselberg iD, Pål Ulleberg, Øystein Sørensen, Knut Inge Fostervold, Sigrid Hegna Ingvaldsen, Ida Svege iD

PLOS One · 2025

DOI: 10.1371/journal.pone.0322696 ↗

Kurz gefasst

Studienprotokoll in der Zeitschrift PLOS One, 2025. Verfasst von Jan-Ole Hesselberg und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Psychology, University of Oslo, Oslo, Norway. Hauptthemen laut MEDLINE: Peer Review, Research, Peer Review, Financing, Organized. Die Arbeit nennt 28 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: EkstraStiftelsen Helse og Rehabilitering.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Grant peer review processes are pivotal in allocating substantial research funding, yet concerns about their reliability persist, primarily due to low inter-rater agreement. This study aims to examine factors associated with agreement among peer reviewers in grant evaluations, leveraging data from 134,991 reviews across four Norwegian research funders. Using a cross-classified linear regression model, we will explore the relationship between inter-rater agreement and multiple factors, including reviewer similarity, experience, expertise, research area, application characteristics, review depth, and temporal trends. Our findings are expected to shed light on whether similarity between reviewers (gender, age), their experience, or expertise correlates with higher agreement. Additionally, we investigate whether characteristics of the applications—such as funding amount, research area, or variability in project size—affect agreement levels. By analyzing applications from diverse disciplines and funding schemes, this study aims to provide a comprehensive understanding of the drivers of inter-rater agreement and their implications for grant peer review reliability. The results will inform improvements to peer review processes, enhancing the fairness and validity of funding decisions. All data and analysis scripts will be publicly available, ensuring transparency and reproducibility.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Peer Review, ResearchPeer ReviewFinancing, OrganizedHumansReproducibility of ResultsNorwayObserver VariationFemaleMalePre-Registration Publication

Autor:innen und Institutionen

  • Department of Psychology, University of Oslo, Oslo, Norway.
  • The Foundation Dam, Oslo, Norway.
  • NIFU Nordic Institute for Studies of Innovation, Research and Education, Oslo, Norway.

ORCID-Profile: Jan-Ole Hesselberg, Ida Svege

Förderung

  • EkstraStiftelsen Helse og Rehabilitering

Zitieren

APA

Hesselberg, J. O., Ulleberg, P., Sørensen, Ø., Fostervold, K. I., Hegna Ingvaldsen, S., & Svege, I. (2025). Registered report protocol: Factors associated with inter-rater agreement in grant peer review. PLOS One. https://doi.org/10.1371/journal.pone.0322696

Vancouver

Hesselberg JO, Ulleberg P, Sørensen Ø, Fostervold KI, Hegna Ingvaldsen S, Svege I. Registered report protocol: Factors associated with inter-rater agreement in grant peer review. PLOS One. 2025. doi:10.1371/journal.pone.0322696

BibTeX
@article{Hesselberg2025,
  author = {Hesselberg, Jan-Ole and Ulleberg, Pål and Sørensen, Øystein and Fostervold, Knut Inge and Hegna Ingvaldsen, Sigrid and Svege, Ida},
  title = {Registered report protocol: Factors associated with inter-rater agreement in grant peer review},
  journal = {PLOS One},
  year = {2025},
  doi = {10.1371/journal.pone.0322696}
}
Kostenlos starten →