Variation and inter‐rater reliability of abstract reviewer scores for clinical vignettes submitted to a national hospitalist meeting
Journal of Hospital Medicine · 2026-05 · 21 (5) · 520-524
DOI: 10.1002/jhm.70261 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Hospital Medicine, online erschienen am 22.01.2026. Verfasst von John R Stephens und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medicine, Division of Hospital Medicine, University of North Carolina School of Medicine, Chapel Hill, USA. Hauptthemen laut MEDLINE: Hospitalists, Peer Review, Research, Congresses as Topic. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 17 Literaturangaben.
Abstract (Originalsprache)
Peer review of research products suffers from poor inter‐rater reliability. Few studies examine whether this limitation generalizes to case reports. We conducted a cross‐sectional analysis of peer reviews of clinical vignette abstracts submitted to a national hospitalist meeting in 2024 and 2025. Three randomly assigned reviewers scored each vignette on a 1–10 scale. We analyzed variation in scores across abstracts and reviewers and estimated inter‐rater reliability via intraclass correlation coefficient (ICC). Two hundred twenty‐one reviewers evaluated 1630 abstracts in 2024–2025. Abstract scores varied substantially: 384/1630 (23.6%) abstracts had a difference of 4 or more points (>2 standard deviations) between highest and lowest reviewer scores. Scores varied by reviewer: 2024 reviewer‐level mean scores ranged 4.27–8.47 (standard deviation (SD): 0.70–2.80); 2025 scores ranged 4.06–8.59 (SD: 0.62‐2.69). Inter‐rater reliability was poor (ICC: 0.37). Adjusting final scores based on reviewer scoring tendencies changed the accept/reject category for 183 (11.2%) abstracts, suggesting opportunities for quality improvement.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Medicine, Division of Hospital Medicine, University of North Carolina School of Medicine, Chapel Hill, USA.
- Departments of Medicine and Pediatrics, Division of Hospital Medicine, Johns Hopkins School of Medicine, Baltimore, Maryland, USA.
ORCID-Profile: Leonard Feldman
Zitieren
APA
Stephens, J. R., Morgan, A., Bellamy, N., Raff, E. J., & Feldman, L. (2026). Variation and inter‐rater reliability of abstract reviewer scores for clinical vignettes submitted to a national hospitalist meeting. Journal of Hospital Medicine, 21(5), 520-524. https://doi.org/10.1002/jhm.70261
Vancouver
Stephens JR, Morgan A, Bellamy N, Raff EJ, Feldman L. Variation and inter‐rater reliability of abstract reviewer scores for clinical vignettes submitted to a national hospitalist meeting. Journal of Hospital Medicine. 2026;21(5):520-524. doi:10.1002/jhm.70261
BibTeX
@article{Stephens2026,
author = {Stephens, John R and Morgan, Andrew and Bellamy, Nelly and Raff, Evan J and Feldman, Leonard},
title = {Variation and inter‐rater reliability of abstract reviewer scores for clinical vignettes submitted to a national hospitalist meeting},
journal = {Journal of Hospital Medicine},
year = {2026},
volume = {21},
number = {5},
pages = {520--524},
doi = {10.1002/jhm.70261}
}
EUVIMED