Correcting for Rater Effects in Operating Room Surgical Skills Assessment
The Laryngoscope · 2024
DOI: 10.1002/lary.31391 ↗Kurz gefasst
Fachartikel in der Zeitschrift The Laryngoscope, 2024. Verfasst von Ryan Chou und 12 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Engineering, Whiting School of Engineering, Johns Hopkins University, Baltimore, Maryland, U.S.A. Hauptthemen laut MEDLINE: Clinical Competence, Operating Rooms, Internship and Residency. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 23 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIDCR NIH HHS, National Institute of Dental & Craniofacial Research of the National Institutes of Health, Provost Undergraduate Research Award from Johns Hopkins University.
Abstract (Originalsprache)
Objective
To estimate and adjust for rater effects in operating room surgical skills assessment performed using a structured rating scale for nasal septoplasty.
Methods
We analyzed survey responses from attending surgeons (raters) who supervised residents and fellows (trainees) performing nasal septoplasty in a prospective cohort study. We fit a structural equation model with the rubric item scores regressed on a latent component of skill and then fit a second model including the rating surgeon as a random effect to model a rater‐effects‐adjusted latent surgical skill. We validated this model against conventional measures including the level of expertise and post‐graduation year (PGY) commensurate with the trainee's performance, the actual PGY of the trainee, and whether the surgical goals were achieved.
Results
Our dataset included 188 assessments by 7 raters and 41 trainees. The model with one latent construct for surgical skill and the rater as a random effect was the best. Rubric scores depended on how severe or lenient the rater was, sometimes almost as much as they depended on trainee skill. Rater‐adjusted latent skill scores increased with attending‐estimated skill levels and PGY of trainees, increased with the actual PGY, and appeared constant over different levels of achievement of surgical goals.
Conclusion
Our work provides a method to obtain rater effect adjusted surgical skill assessments in the operating room using structured rating scales. Our method allows for the creation of standardized (i.e., rater‐effects‐adjusted) quantitative surgical skill benchmarks using national‐level databases on trainee assessments. Level of Evidence N/A Laryngoscope , 134:3548–3554, 2024
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biomedical Engineering, Whiting School of Engineering, Johns Hopkins University, Baltimore, Maryland, U.S.A.
- Dugoni School of Dentistry, University of Pacific, San Francisco, California, U.S.A.
- Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A.
- Division of Facial Plastic and Reconstructive Surgery, Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins University School of Medicine, Baltimore, Maryland, U.S.A.
- Department of Otolaryngology-Head and Neck Surgery, MedStar Georgetown University Hospital, Washington, DC, U.S.A.
- ENT Section, Veterans Affairs Medical Center, Washington, DC, U.S.A.
ORCID-Profile: Ryan Chou, Kofi D O Boahene
Förderung
- NIDCR NIH HHS
- National Institute of Dental & Craniofacial Research of the National Institutes of Health
- Provost Undergraduate Research Award from Johns Hopkins University
Zitieren
APA
Chou, R., Naz, H., Boahene, K. D. O., Maxwell, J. H., Wanamaker, J. R., Byrne, P. J., Papel, I. D., Kontis, T. C., Hager, G. D., Ishii, L. E., Malekzadeh, S., Vedula, S. S., & Ishii, M. (2024). Correcting for Rater Effects in Operating Room Surgical Skills Assessment. The Laryngoscope. https://doi.org/10.1002/lary.31391
Vancouver
Chou R, Naz H, Boahene KDO, Maxwell JH, Wanamaker JR, Byrne PJ, et al. Correcting for Rater Effects in Operating Room Surgical Skills Assessment. The Laryngoscope. 2024. doi:10.1002/lary.31391
BibTeX
@article{Chou2024,
author = {Chou, Ryan and Naz, Hajira and Boahene, Kofi D O and Maxwell, Jessica H and Wanamaker, John R and Byrne, Patrick J and Papel, Ira D and Kontis, Theda C and Hager, Gregory D and Ishii, Lisa E and Malekzadeh, Sonya and Vedula, S Swaroop and Ishii, Masaru},
title = {Correcting for Rater Effects in Operating Room Surgical Skills Assessment},
journal = {The Laryngoscope},
year = {2024},
doi = {10.1002/lary.31391}
}
EUVIMED