EUVIMEDEUVIMED entdecken
Validierungsstudie

Validity and reliability of Brier scoring for assessment of probabilistic diagnostic reasoning

Nathan Stehouwer iD, Anastasia Rowland-Seymour, Larry Gruppen, Jeffrey M Albert, Kelli Qua

Diagnosis · 20.02.2025 · 12 (1) · 53-60

DOI: 10.1515/dx-2023-0109 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Diagnosis, online erschienen am 16.10.2024. Verfasst von Nathan Stehouwer und 4 weiteren Autor:innen. Beteiligte Einrichtung: University Hospitals Cleveland Medical Center and Rainbow Babies & Children's Hospital, Cleveland, OH, USA. Hauptthemen laut MEDLINE: Clinical Reasoning, Clinical Competence, Educational Measurement. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 29 Literaturangaben. Gefördert von: University Hospitals Graduate Medical Education, Zucker Neurology Fund.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Educators need tools for the assessment of clinical reasoning that reflect the ambiguity of real-world practice and measure learners’ ability to determine diagnostic likelihood. In this study, the authors describe the use of the Brier score to assess and provide feedback on the quality of probabilistic diagnostic reasoning.

Methods

The authors describe a novel format called Diagnostic Forecasting (DxF), in which participants read a brief clinical case and assign a probability to each item on a differential diagnosis, order tests and select a final diagnosis. DxF was piloted in a cohort of senior medical students. DxF evaluated students’ answers with Brier scores, which compare probabilistic forecasts with case outcomes. The validity of Brier scores in DxF was assessed by comparison to subsequent decision-making in the game environment of DxF, as well as external criteria including medical knowledge tests and performance on clinical rotations.

Results

Brier scores were statistically significantly correlated with diagnostic accuracy (95 % CI −4.4 to −0.44) and with mean scores on the National Board of Medical Examiners (NBME) shelf exams (95 % CI −474.6 to −225.1). Brier scores did not correlate with clerkship grades or performance on a structured clinical skills exam. Reliability as measured by within-student correlation was low.

Conclusions

Brier scoring showed evidence for validity as a measurement of medical knowledge and predictor of clinical decision-making. Further work must evaluated the ability of Brier scores to predict clinical and workplace-based outcomes, and develop reliable approaches to measuring probabilistic reasoning.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Clinical ReasoningClinical CompetenceEducational MeasurementHumansReproducibility of ResultsStudents, MedicalProbabilityDiagnosis, DifferentialClinical Decision-Making

Stichwörter der Autor:innen

assessmentdiagnostic reasoningprobabilistic reasoninguncertainty

Autor:innen und Institutionen

  • University Hospitals Cleveland Medical Center and Rainbow Babies & Children's Hospital, Cleveland, OH, USA.
  • Case Western Reserve University School of Medicine, Cleveland, OH, USA.
  • MetroHealth Medical Center, Cleveland, OH, USA.
  • University of Michigan Medical School, Ann Arbor, MI, USA.

ORCID-Profile: Nathan Stehouwer

Förderung

  • University Hospitals Graduate Medical Education
  • Zucker Neurology Fund

Zitieren

APA

Stehouwer, N., Rowland-Seymour, A., Gruppen, L., Albert, J. M., & Qua, K. (2025). Validity and reliability of Brier scoring for assessment of probabilistic diagnostic reasoning. Diagnosis, 12(1), 53-60. https://doi.org/10.1515/dx-2023-0109

Vancouver

Stehouwer N, Rowland-Seymour A, Gruppen L, Albert JM, Qua K. Validity and reliability of Brier scoring for assessment of probabilistic diagnostic reasoning. Diagnosis. 2025;12(1):53-60. doi:10.1515/dx-2023-0109

BibTeX
@article{Stehouwer2025,
  author = {Stehouwer, Nathan and Rowland-Seymour, Anastasia and Gruppen, Larry and Albert, Jeffrey M and Qua, Kelli},
  title = {Validity and reliability of Brier scoring for assessment of probabilistic diagnostic reasoning},
  journal = {Diagnosis},
  year = {2025},
  volume = {12},
  number = {1},
  pages = {53--60},
  doi = {10.1515/dx-2023-0109}
}
Kostenlos starten →