Validity and reliability of Brier scoring for assessment of probabilistic diagnostic reasoning
Diagnosis · 20.02.2025 · 12 (1) · 53-60
DOI: 10.1515/dx-2023-0109 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift Diagnosis, online erschienen am 16.10.2024. Verfasst von Nathan Stehouwer und 4 weiteren Autor:innen. Beteiligte Einrichtung: University Hospitals Cleveland Medical Center and Rainbow Babies & Children's Hospital, Cleveland, OH, USA. Hauptthemen laut MEDLINE: Clinical Reasoning, Clinical Competence, Educational Measurement. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 29 Literaturangaben. Gefördert von: University Hospitals Graduate Medical Education, Zucker Neurology Fund.
Abstract (Originalsprache)
Objectives
Educators need tools for the assessment of clinical reasoning that reflect the ambiguity of real-world practice and measure learners’ ability to determine diagnostic likelihood. In this study, the authors describe the use of the Brier score to assess and provide feedback on the quality of probabilistic diagnostic reasoning.
Methods
The authors describe a novel format called Diagnostic Forecasting (DxF), in which participants read a brief clinical case and assign a probability to each item on a differential diagnosis, order tests and select a final diagnosis. DxF was piloted in a cohort of senior medical students. DxF evaluated students’ answers with Brier scores, which compare probabilistic forecasts with case outcomes. The validity of Brier scores in DxF was assessed by comparison to subsequent decision-making in the game environment of DxF, as well as external criteria including medical knowledge tests and performance on clinical rotations.
Results
Brier scores were statistically significantly correlated with diagnostic accuracy (95 % CI −4.4 to −0.44) and with mean scores on the National Board of Medical Examiners (NBME) shelf exams (95 % CI −474.6 to −225.1). Brier scores did not correlate with clerkship grades or performance on a structured clinical skills exam. Reliability as measured by within-student correlation was low.
Conclusions
Brier scoring showed evidence for validity as a measurement of medical knowledge and predictor of clinical decision-making. Further work must evaluated the ability of Brier scores to predict clinical and workplace-based outcomes, and develop reliable approaches to measuring probabilistic reasoning.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- University Hospitals Cleveland Medical Center and Rainbow Babies & Children's Hospital, Cleveland, OH, USA.
- Case Western Reserve University School of Medicine, Cleveland, OH, USA.
- MetroHealth Medical Center, Cleveland, OH, USA.
- University of Michigan Medical School, Ann Arbor, MI, USA.
ORCID-Profile: Nathan Stehouwer
Förderung
- University Hospitals Graduate Medical Education
- Zucker Neurology Fund
Zitieren
APA
Stehouwer, N., Rowland-Seymour, A., Gruppen, L., Albert, J. M., & Qua, K. (2025). Validity and reliability of Brier scoring for assessment of probabilistic diagnostic reasoning. Diagnosis, 12(1), 53-60. https://doi.org/10.1515/dx-2023-0109
Vancouver
Stehouwer N, Rowland-Seymour A, Gruppen L, Albert JM, Qua K. Validity and reliability of Brier scoring for assessment of probabilistic diagnostic reasoning. Diagnosis. 2025;12(1):53-60. doi:10.1515/dx-2023-0109
BibTeX
@article{Stehouwer2025,
author = {Stehouwer, Nathan and Rowland-Seymour, Anastasia and Gruppen, Larry and Albert, Jeffrey M and Qua, Kelli},
title = {Validity and reliability of Brier scoring for assessment of probabilistic diagnostic reasoning},
journal = {Diagnosis},
year = {2025},
volume = {12},
number = {1},
pages = {53--60},
doi = {10.1515/dx-2023-0109}
}
EUVIMED