EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Inconsistencies in rater-based assessments mainly affect borderline candidates: but using simple heuristics might improve pass-fail decisions

Stefan K Schauber, Anne O Olsen, Erik L Werner, Morten Magelssen

Advances in Health Sciences Education · 2024

DOI: 10.1007/s10459-024-10328-0 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Advances in Health Sciences Education, 2024. Verfasst von Stefan K Schauber und 3 weiteren Autor:innen. Beteiligte Einrichtung: Centre for Health Sciences Education, Faculty of Medicine, University of Oslo, Oslo, Norway. stefanks@uio.no. Hauptthemen laut MEDLINE: Heuristics, Educational Measurement, Decision Making. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: University of Oslo.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

Research in various areas indicates that expert judgment can be highly inconsistent. However, expert judgment is indispensable in many contexts. In medical education, experts often function as examiners in rater-based assessments. Here, disagreement between examiners can have far-reaching consequences. The literature suggests that inconsistencies in ratings depend on the level of performance a to-be-evaluated candidate shows. This possibility has not been addressed deliberately and with appropriate statistical methods. By adopting the theoretical lens of ecological rationality, we evaluate if easily implementable strategies can enhance decision making in real-world assessment contexts.

Methods

We address two objectives. First, we investigate the dependence of rater-consistency on performance levels. We recorded videos of mock-exams and had examiners (N=10) evaluate four students’ performances and compare inconsistencies in performance ratings between examiner-pairs using a bootstrapping procedure. Our second objective is to provide an approach that aids decision making by implementing simple heuristics.

Results

We found that discrepancies were largely a function of the level of performance the candidates showed. Lower performances were rated more inconsistently than excellent performances. Furthermore, our analyses indicated that the use of simple heuristics might improve decisions in examiner pairs.

Discussion

Inconsistencies in performance judgments continue to be a matter of concern, and we provide empirical evidence for them to be related to candidate performance. We discuss implications for research and the advantages of adopting the perspective of ecological rationality. We point to directions both for further research and for development of assessment practices.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

HeuristicsEducational MeasurementDecision MakingHumansJudgmentClinical CompetenceStudents, MedicalEducation, MedicalObserver Variation

Stichwörter der Autor:innen

Borderline candidatesExpert judgmentHeuristicsRater inconsistencyRater-based assessment

Autor:innen und Institutionen

  • Centre for Health Sciences Education, Faculty of Medicine, University of Oslo, Oslo, Norway. stefanks@uio.no.
  • Centre for Educational Measurement (CEMO), Faculty of Educational Sciences, University of Oslo, Oslo, Norway. stefanks@uio.no.
  • Department of Community Medicine and Global Health, Institute of Health and Society, University of Oslo, Oslo, Norway.
  • Department of General Practice, Institute of Health and Society, University of Oslo, Oslo, Norway.
  • Centre for Medical Ethics, Institute of Health and Society, University of Oslo, Oslo, Norway.

Förderung

  • University of Oslo

Zitieren

APA

Schauber, S. K., Olsen, A. O., Werner, E. L., & Magelssen, M. (2024). Inconsistencies in rater-based assessments mainly affect borderline candidates: but using simple heuristics might improve pass-fail decisions. Advances in Health Sciences Education. https://doi.org/10.1007/s10459-024-10328-0

Vancouver

Schauber SK, Olsen AO, Werner EL, Magelssen M. Inconsistencies in rater-based assessments mainly affect borderline candidates: but using simple heuristics might improve pass-fail decisions. Advances in Health Sciences Education. 2024. doi:10.1007/s10459-024-10328-0

BibTeX
@article{Schauber2024,
  author = {Schauber, Stefan K and Olsen, Anne O and Werner, Erik L and Magelssen, Morten},
  title = {Inconsistencies in rater-based assessments mainly affect borderline candidates: but using simple heuristics might improve pass-fail decisions},
  journal = {Advances in Health Sciences Education},
  year = {2024},
  doi = {10.1007/s10459-024-10328-0}
}
Kostenlos starten →