Automated Evaluation of Reflection and Feedback Quality in Workplace-Based Assessments by Using Natural Language Processing: Cross-Sectional Competency-Based Medical Education Study
JMIR Medical Education · 2025
DOI: 10.2196/81718 ↗Kurz gefasst
Querschnittstudie in der Zeitschrift JMIR Medical Education, 2025. Verfasst von Jeng-Wen Chen und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Otolaryngology-Head and Neck Surgery, Cardinal Tien Hospital, Fu Jen Catholic University, New Taipei City, Taiwan. Hauptthemen laut MEDLINE: Natural Language Processing, Competency-Based Education, Clinical Competence, Educational Measurement. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Competency-based medical education relies heavily on high-quality narrative reflections and feedback within workplace-based assessments. However, evaluating these narratives at scale remains a significant challenge.
Objective
This study aims to develop and apply natural language processing (NLP) models to evaluate the quality of resident reflections and faculty feedback documented in Entrustable Professional Activities (EPAs) on Taiwan’s nationwide Emyway platform for otolaryngology residency training.
Methods
This 4-year cross-sectional study analyzes 300 randomly sampled EPA assessments from 2021 to 2025, covering a pilot year and 3 full implementation years. Two medical education experts independently rated the narratives based on relevance, specificity, and the presence of reflective or improvement-focused language. Narratives were categorized into 4 quality levels—effective, moderate, ineffective, or irrelevant—and then dichotomized into high quality and low quality. We compared the performance of logistic regression, support vector machine, and bidirectional encoder representations from transformers (BERT) models in classifying narrative quality. The best performing model was then applied to track quality trends over time.
Results
The BERT model, a multilingual pretrained language model, outperformed other approaches, achieving 85% and 92% accuracy in binary classification for resident reflections and faculty feedback, respectively. The accuracy for the 4-level classification was 67% for both. Longitudinal analysis revealed significant increases in high-quality reflections (from 70.3% to 99.5%) and feedback (from 50.6% to 88.9%) over the study period.
Conclusions
BERT-based NLP demonstrated moderate-to-high accuracy in evaluating the narrative quality in EPA assessments, especially in the binary classification. While not a replacement for expert review, NLP models offer a valuable tool for monitoring narrative trends and enhancing formative feedback in competency-based medical education.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Otolaryngology-Head and Neck Surgery, Cardinal Tien Hospital, Fu Jen Catholic University, New Taipei City, Taiwan.
- Department of Otolaryngology-Head and Neck Surgery, National Taiwan University Hospital and Children's Hospital, Taipei, Taiwan.
- Department of Hospital Management, Graduate Institute of Business Administration, Fu Jen Catholic University, New Taipei City, Taiwan.
- Department of Education and Research, Cardinal Tien Junior College of Healthcare and Management, New Taipei City, Taiwan.
- Department of Library and Information Science, Fu-Jen Catholic University, New Taipei City, Taiwan.
- Cathay General Hospital, Department of Otolaryngology, Taipei, Taiwan.
ORCID-Profile: Jeng-Wen Chen, Hai-Lun Tu, Chun-Hsiang Chang, Wei-Chung Hsu, Pa-Chun Wang, Chun-Hou Liao, Mingchih Chen
Zitieren
APA
Chen, J. W., Tu, H. L., Chang, C. H., Hsu, W. C., Wang, P. C., Liao, C. H., & Chen, M. (2025). Automated Evaluation of Reflection and Feedback Quality in Workplace-Based Assessments by Using Natural Language Processing: Cross-Sectional Competency-Based Medical Education Study. JMIR Medical Education. https://doi.org/10.2196/81718
Vancouver
Chen JW, Tu HL, Chang CH, Hsu WC, Wang PC, Liao CH, et al. Automated Evaluation of Reflection and Feedback Quality in Workplace-Based Assessments by Using Natural Language Processing: Cross-Sectional Competency-Based Medical Education Study. JMIR Medical Education. 2025. doi:10.2196/81718
BibTeX
@article{Chen2025,
author = {Chen, Jeng-Wen and Tu, Hai-Lun and Chang, Chun-Hsiang and Hsu, Wei-Chung and Wang, Pa-Chun and Liao, Chun-Hou and Chen, Mingchih},
title = {Automated Evaluation of Reflection and Feedback Quality in Workplace-Based Assessments by Using Natural Language Processing: Cross-Sectional Competency-Based Medical Education Study},
journal = {JMIR Medical Education},
year = {2025},
doi = {10.2196/81718}
}
EUVIMED