EUVIMEDEUVIMED entdecken
BeobachtungsstudieOpen Access · CC BY 4.0

Reliability of a workplace-based assessment for the United States general surgical trainees’ intraoperative performance using multivariate generalizability theory: a psychometric study

Ting Sun, Stella Yun Kim, Brigitte Kristin Smith, Yoon Soo Park

Journal of Educational Evaluation for Health Professions · 2024

DOI: 10.3352/jeehp.2024.21.26 ↗

Kurz gefasst

Beobachtungsstudie in der Zeitschrift Journal of Educational Evaluation for Health Professions, 2024. Verfasst von Ting Sun und 3 weiteren Autor:innen. Beteiligte Einrichtung: Office of Surgical Education, Department of Surgery, University of Utah, Salt Lake City, UT, USA. Hauptthemen laut MEDLINE: Clinical Competence, Educational Measurement, General Surgery, Internship and Residency. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 20 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

The System for Improving and Measuring Procedure Learning (SIMPL), a smartphone-based operative assessment application, was developed to assess the intraoperative performance of surgical residents. This study aims to examine the reliability of the SIMPL assessment and determine the optimal number of procedures for a reliable assessment.

Methods

In this retrospective observational study, we analyzed data collected between 2015 and 2023 from 4,616 residents across 94 General Surgery Residency programs in the United States that utilized the SIMPL smartphone application. We employed multivariate generalizability theory and initially conducted generalizability studies to estimate the variance components associated with procedures. We then performed decision studies to estimate the reliability coefficient and the minimum number of procedures required for a reproducible assessment.

Results

We estimated that the reliability of the assessment of surgical trainees’ intraoperative autonomy and performance using SIMPL exceeded 0.70. Additionally, the optimal number of procedures required for a reproducible assessment was 10, 17, 15, and 17 for postgraduate year (PGY) 2, PGY 3, PGY 4, and PGY 5, respectively. Notably, the study highlighted that the assessment of residents in their senior years necessitated a larger number of procedures compared to those in their junior years.

Conclusion

The study demonstrated that the SIMPL assessment is reliably effective for evaluating the intraoperative performance of surgical trainees. Adjusting the number of procedures based on the trainees’ training stage enhances the assessment process’s accuracy and effectiveness.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Clinical CompetenceEducational MeasurementGeneral SurgeryInternship and ResidencyPsychometricsSmartphoneFemaleHumansMaleEducation, Medical, GraduateMobile ApplicationsReproducibility of ResultsRetrospective StudiesUnited StatesWorkplace

Stichwörter der Autor:innen

Internship and residencyMobile applicationsTraining supportUnited StatesSmartphone

Autor:innen und Institutionen

  • Office of Surgical Education, Department of Surgery, University of Utah, Salt Lake City, UT, USA.
  • Department of Educational Leadership, University of North Carolina at Charlotte, Charlotte, NC, USA.
  • Division of Vascular Surgery, Department of Surgery, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA.
  • Department of Medical Education, University of Illinois Chicago, Chicago, IN, USA.

Zitieren

APA

Sun, T., Kim, S. Y., Smith, B. K., & Park, Y. S. (2024). Reliability of a workplace-based assessment for the United States general surgical trainees’ intraoperative performance using multivariate generalizability theory: a psychometric study. Journal of Educational Evaluation for Health Professions. https://doi.org/10.3352/jeehp.2024.21.26

Vancouver

Sun T, Kim SY, Smith BK, Park YS. Reliability of a workplace-based assessment for the United States general surgical trainees’ intraoperative performance using multivariate generalizability theory: a psychometric study. Journal of Educational Evaluation for Health Professions. 2024. doi:10.3352/jeehp.2024.21.26

BibTeX
@article{Sun2024,
  author = {Sun, Ting and Kim, Stella Yun and Smith, Brigitte Kristin and Park, Yoon Soo},
  title = {Reliability of a workplace-based assessment for the United States general surgical trainees’ intraoperative performance using multivariate generalizability theory: a psychometric study},
  journal = {Journal of Educational Evaluation for Health Professions},
  year = {2024},
  doi = {10.3352/jeehp.2024.21.26}
}
Kostenlos starten →