How well do workplace‐based assessments support summative entrustment decisions? A multi‐institutional generalisability study
Medical Education · 2024
DOI: 10.1111/medu.15291 ↗Kurz gefasst
Multizentrische Studie in der Zeitschrift Medical Education, 2024. Verfasst von Michael S Ryan und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Pediatrics, University of Virginia, Charlottesville, Virginia, USA. Hauptthemen laut MEDLINE: Educational Measurement, Workplace, Clinical Competence. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 54 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Association of American Medical Colleges, Southern Group on Educational Affairs.
Abstract (Originalsprache)
Background
Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace‐based assessments (WBAs). Single‐institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi‐institutional generalisability study to determine the influence of specific factors on reliability of WBAs.
Methods
The authors analysed WBA data obtained for clerkship‐level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross‐classified random‐effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability.
Results
Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 ( n = 17 118, 32%) 2 ( n = 10 237, 19.1%), and 6 ( n = 6000, 18.5%). Low variance attributed to the learner (560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability.
Conclusion
The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high‐stakes summative entrustment decisions.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Pediatrics, University of Virginia, Charlottesville, Virginia, USA.
- School of Health Professions Education, Maastricht University, Maastricht, The Netherlands.
- Departments of Medicine and Pediatrics, Emory University School of Medicine, Atlanta, Georgia, USA.
- Department of Biostatistics, Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA.
- Departments of Pediatrics, Biomedical Education and Data Science, Lewis Katz School of Medicine, Philadelphia, Pennsylvania, USA.
- Department of Medicine, College of Human Medicine, Michigan State University, East Lansing, Michigan, USA.
ORCID-Profile: Michael S Ryan, Gary Ferenchick
Förderung
- Association of American Medical Colleges
- Southern Group on Educational Affairs
Zitieren
APA
Ryan, M. S., Gielissen, K. A., Shin, D., Perera, R. A., Gusic, M., Ferenchick, G., Ownby, A., Cutrer, W. B., Obeso, V., & Santen, S. A. (2024). How well do workplace‐based assessments support summative entrustment decisions? A multi‐institutional generalisability study. Medical Education. https://doi.org/10.1111/medu.15291
Vancouver
Ryan MS, Gielissen KA, Shin D, Perera RA, Gusic M, Ferenchick G, et al. How well do workplace‐based assessments support summative entrustment decisions? A multi‐institutional generalisability study. Medical Education. 2024. doi:10.1111/medu.15291
BibTeX
@article{Ryan2024,
author = {Ryan, Michael S and Gielissen, Katherine A and Shin, Dongho and Perera, Robert A and Gusic, Maryellen and Ferenchick, Gary and Ownby, Allison and Cutrer, William B and Obeso, Vivian and Santen, Sally A},
title = {How well do workplace‐based assessments support summative entrustment decisions? A multi‐institutional generalisability study},
journal = {Medical Education},
year = {2024},
doi = {10.1111/medu.15291}
}
EUVIMED