EUVIMEDEUVIMED entdecken
Multizentrische StudieOpen Access · CC BY-NC 4.0

How well do workplace‐based assessments support summative entrustment decisions? A multi‐institutional generalisability study

Michael S Ryan iD, Katherine A Gielissen, Dongho Shin, Robert A Perera, Maryellen Gusic, Gary Ferenchick iD, Allison Ownby, William B Cutrer, Vivian Obeso, Sally A Santen

Medical Education · 2024

DOI: 10.1111/medu.15291 ↗

Kurz gefasst

Multizentrische Studie in der Zeitschrift Medical Education, 2024. Verfasst von Michael S Ryan und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Pediatrics, University of Virginia, Charlottesville, Virginia, USA. Hauptthemen laut MEDLINE: Educational Measurement, Workplace, Clinical Competence. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 54 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Association of American Medical Colleges, Southern Group on Educational Affairs.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace‐based assessments (WBAs). Single‐institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi‐institutional generalisability study to determine the influence of specific factors on reliability of WBAs.

Methods

The authors analysed WBA data obtained for clerkship‐level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross‐classified random‐effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability.

Results

Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 ( n = 17 118, 32%) 2 ( n = 10 237, 19.1%), and 6 ( n = 6000, 18.5%). Low variance attributed to the learner (560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability.

Conclusion

The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high‐stakes summative entrustment decisions.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Educational MeasurementWorkplaceClinical CompetenceHumansReproducibility of ResultsStudents, MedicalCompetency-Based EducationInternship and ResidencyClinical Clerkship

Autor:innen und Institutionen

  • Department of Pediatrics, University of Virginia, Charlottesville, Virginia, USA.
  • School of Health Professions Education, Maastricht University, Maastricht, The Netherlands.
  • Departments of Medicine and Pediatrics, Emory University School of Medicine, Atlanta, Georgia, USA.
  • Department of Biostatistics, Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA.
  • Departments of Pediatrics, Biomedical Education and Data Science, Lewis Katz School of Medicine, Philadelphia, Pennsylvania, USA.
  • Department of Medicine, College of Human Medicine, Michigan State University, East Lansing, Michigan, USA.

ORCID-Profile: Michael S Ryan, Gary Ferenchick

Förderung

  • Association of American Medical Colleges
  • Southern Group on Educational Affairs

Zitieren

APA

Ryan, M. S., Gielissen, K. A., Shin, D., Perera, R. A., Gusic, M., Ferenchick, G., Ownby, A., Cutrer, W. B., Obeso, V., & Santen, S. A. (2024). How well do workplace‐based assessments support summative entrustment decisions? A multi‐institutional generalisability study. Medical Education. https://doi.org/10.1111/medu.15291

Vancouver

Ryan MS, Gielissen KA, Shin D, Perera RA, Gusic M, Ferenchick G, et al. How well do workplace‐based assessments support summative entrustment decisions? A multi‐institutional generalisability study. Medical Education. 2024. doi:10.1111/medu.15291

BibTeX
@article{Ryan2024,
  author = {Ryan, Michael S and Gielissen, Katherine A and Shin, Dongho and Perera, Robert A and Gusic, Maryellen and Ferenchick, Gary and Ownby, Allison and Cutrer, William B and Obeso, Vivian and Santen, Sally A},
  title = {How well do workplace‐based assessments support summative entrustment decisions? A multi‐institutional generalisability study},
  journal = {Medical Education},
  year = {2024},
  doi = {10.1111/medu.15291}
}
Kostenlos starten →