EUVIMEDEUVIMED entdecken
QuerschnittstudieOpen Access

Automated Evaluation of Reflection and Feedback Quality in Workplace-Based Assessments by Using Natural Language Processing: Cross-Sectional Competency-Based Medical Education Study

Jeng-Wen Chen iD, Hai-Lun Tu iD, Chun-Hsiang Chang iD, Wei-Chung Hsu iD, Pa-Chun Wang iD, Chun-Hou Liao iD, Mingchih Chen iD

JMIR Medical Education · 2025

DOI: 10.2196/81718 ↗

Kurz gefasst

Querschnittstudie in der Zeitschrift JMIR Medical Education, 2025. Verfasst von Jeng-Wen Chen und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Otolaryngology-Head and Neck Surgery, Cardinal Tien Hospital, Fu Jen Catholic University, New Taipei City, Taiwan. Hauptthemen laut MEDLINE: Natural Language Processing, Competency-Based Education, Clinical Competence, Educational Measurement. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Competency-based medical education relies heavily on high-quality narrative reflections and feedback within workplace-based assessments. However, evaluating these narratives at scale remains a significant challenge.

Objective

This study aims to develop and apply natural language processing (NLP) models to evaluate the quality of resident reflections and faculty feedback documented in Entrustable Professional Activities (EPAs) on Taiwan’s nationwide Emyway platform for otolaryngology residency training.

Methods

This 4-year cross-sectional study analyzes 300 randomly sampled EPA assessments from 2021 to 2025, covering a pilot year and 3 full implementation years. Two medical education experts independently rated the narratives based on relevance, specificity, and the presence of reflective or improvement-focused language. Narratives were categorized into 4 quality levels—effective, moderate, ineffective, or irrelevant—and then dichotomized into high quality and low quality. We compared the performance of logistic regression, support vector machine, and bidirectional encoder representations from transformers (BERT) models in classifying narrative quality. The best performing model was then applied to track quality trends over time.

Results

The BERT model, a multilingual pretrained language model, outperformed other approaches, achieving 85% and 92% accuracy in binary classification for resident reflections and faculty feedback, respectively. The accuracy for the 4-level classification was 67% for both. Longitudinal analysis revealed significant increases in high-quality reflections (from 70.3% to 99.5%) and feedback (from 50.6% to 88.9%) over the study period.

Conclusions

BERT-based NLP demonstrated moderate-to-high accuracy in evaluating the narrative quality in EPA assessments, especially in the binary classification. While not a replacement for expert review, NLP models offer a valuable tool for monitoring narrative trends and enhancing formative feedback in competency-based medical education.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingCompetency-Based EducationClinical CompetenceEducational MeasurementOtolaryngologyCross-Sectional StudiesHumansTaiwanInternship and ResidencyWorkplaceFeedback

Stichwörter der Autor:innen

Emyway platformcompetency-based medical educationentrustable professional activitiesfeedbackotolaryngologyreflectionresidencyworkplace-based assessment

Autor:innen und Institutionen

  • Department of Otolaryngology-Head and Neck Surgery, Cardinal Tien Hospital, Fu Jen Catholic University, New Taipei City, Taiwan.
  • Department of Otolaryngology-Head and Neck Surgery, National Taiwan University Hospital and Children's Hospital, Taipei, Taiwan.
  • Department of Hospital Management, Graduate Institute of Business Administration, Fu Jen Catholic University, New Taipei City, Taiwan.
  • Department of Education and Research, Cardinal Tien Junior College of Healthcare and Management, New Taipei City, Taiwan.
  • Department of Library and Information Science, Fu-Jen Catholic University, New Taipei City, Taiwan.
  • Cathay General Hospital, Department of Otolaryngology, Taipei, Taiwan.

ORCID-Profile: Jeng-Wen Chen, Hai-Lun Tu, Chun-Hsiang Chang, Wei-Chung Hsu, Pa-Chun Wang, Chun-Hou Liao, Mingchih Chen

Zitieren

APA

Chen, J. W., Tu, H. L., Chang, C. H., Hsu, W. C., Wang, P. C., Liao, C. H., & Chen, M. (2025). Automated Evaluation of Reflection and Feedback Quality in Workplace-Based Assessments by Using Natural Language Processing: Cross-Sectional Competency-Based Medical Education Study. JMIR Medical Education. https://doi.org/10.2196/81718

Vancouver

Chen JW, Tu HL, Chang CH, Hsu WC, Wang PC, Liao CH, et al. Automated Evaluation of Reflection and Feedback Quality in Workplace-Based Assessments by Using Natural Language Processing: Cross-Sectional Competency-Based Medical Education Study. JMIR Medical Education. 2025. doi:10.2196/81718

BibTeX
@article{Chen2025,
  author = {Chen, Jeng-Wen and Tu, Hai-Lun and Chang, Chun-Hsiang and Hsu, Wei-Chung and Wang, Pa-Chun and Liao, Chun-Hou and Chen, Mingchih},
  title = {Automated Evaluation of Reflection and Feedback Quality in Workplace-Based Assessments by Using Natural Language Processing: Cross-Sectional Competency-Based Medical Education Study},
  journal = {JMIR Medical Education},
  year = {2025},
  doi = {10.2196/81718}
}
Kostenlos starten →