EUVIMEDEUVIMED entdecken
Crossref

Challenges in using natural language processing to stratify students by narrative assessments in undergraduate medical education

Laurah Turner iD, Christine Yang Zhou iD, Danielle Weber iD, Seth Overla iD, Mathew Kelleher iD, Pamela Baker iD, Sally A Santen iD

Academic Medicine · 2025

DOI: 10.1093/acamed/wvaf065 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Academic Medicine, 2025. Verfasst von Laurah Turner und 6 weiteren Autor:innen. Beteiligte Einrichtung: Artificial Intelligence and Educational Informatics and Department of Medical Education, University of Cincinnati College of Medicine, Cincinnati, OH…. Hauptthemen laut MEDLINE: Natural Language Processing, Education, Medical, Undergraduate, Narration, Educational Measurement. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 10 Literaturangaben. Gefördert von: CTSA, NCATS NIH HHS, National Center for Advancing Translational Sciences.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

PROBLEM: Medical educators pay increasing attention to the potential utility of narrative data for assessment, but lack of efficient and standardized ways of interpreting the data have limited its use. Natural language processing (NLP) algorithms could provide new insights for using narrative data within assessment especially identifying at-risk or low-performing students. APPROACH: Assessment data were reviewed from 16 cohorts of medical students from the University of Cincinnati College of Medicine (graduating classes of 2006-2022). A T-score Average (TSA) was calculated for each student based on clerkship assessment data. Narrative data from core clerkship evaluations in responses to the prompt "any opportunities for improvement" were utilized for analysis. The narrative data and calculated TSA were then used to train and test 4 NLP models with the goal of utilizing NLP to identify at-risk students as defined by the bottom 10% average TSA score. OUTCOMES: Based on typical NLP methods, the developed NLP models all performed adequately in identifying student performance with an overall accuracy of 0.8 across all 4 models. However, none of the NLP models was able to identify students within the bottom 10% of performance. During this process, we uncovered the presence of "copy/paste" behavior, a previously undocumented phenomenon within narrative data where preceptors duplicated comments from 1 student to the next. Training NLP models including "copy/paste" comments improved NLP ability to identify students within the bottom and top 10% of performance. NEXT STEPS: NLP models were unable to accurately identify at-risk students. Model accuracy increased with the inclusion of "copy/paste" comments, indicating there may be some discriminatory functionality within aspects of narrative data beyond keywords such as lexical diversity and word quantity. Future work will explore how these other aspects of narrative data associate with performance and utilize more advanced large language models of analysis.Based on typical natural language processing (NLP) methods, the tested NLP models performed adequately in identifying student performance with an overall accuracy of 0.8 across all models. However, none of the NLP models was able to identify students within the bottom 10% of performance, revealing widespread "copy/paste" behavior in evaluations that may limit narrative data's discriminatory utility.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingEducation, Medical, UndergraduateNarrationEducational MeasurementStudents, MedicalHumansClinical ClerkshipOhioAlgorithms

Stichwörter der Autor:innen

assessmentmedical studentnarrative assessmentnatural language processing

Autor:innen und Institutionen

  • Artificial Intelligence and Educational Informatics and Department of Medical Education, University of Cincinnati College of Medicine, Cincinnati, OH, United States.
  • Division of Pulmonary and Critical Care, University of Cincinnati College of Medicine, Cincinnati, OH, United States.
  • Internal Medicine and Pediatrics, Departments of Pediatrics and Internal Medicine, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, OH, United States.
  • Office of Medical Education, University of Cincinnati College of Medicine, Cincinnati, OH, United States.
  • Internal Medicine and Pediatrics, University of Cincinnati College of Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States.
  • Medical Education and Department of Medical Education, University of Cincinnati, Cincinnati, OH, United States.

ORCID-Profile: Laurah Turner, Christine Yang Zhou, Danielle Weber, Seth Overla, Mathew Kelleher, Pamela Baker, Sally A Santen

Förderung

  • CTSA
  • NCATS NIH HHS
  • National Center for Advancing Translational Sciences

Zitieren

APA

Turner, L., Zhou, C. Y., Weber, D., Overla, S., Kelleher, M., Baker, P., & Santen, S. A. (2025). Challenges in using natural language processing to stratify students by narrative assessments in undergraduate medical education. Academic Medicine. https://doi.org/10.1093/acamed/wvaf065

Vancouver

Turner L, Zhou CY, Weber D, Overla S, Kelleher M, Baker P, et al. Challenges in using natural language processing to stratify students by narrative assessments in undergraduate medical education. Academic Medicine. 2025. doi:10.1093/acamed/wvaf065

BibTeX
@article{Turner2025,
  author = {Turner, Laurah and Zhou, Christine Yang and Weber, Danielle and Overla, Seth and Kelleher, Mathew and Baker, Pamela and Santen, Sally A},
  title = {Challenges in using natural language processing to stratify students by narrative assessments in undergraduate medical education},
  journal = {Academic Medicine},
  year = {2025},
  doi = {10.1093/acamed/wvaf065}
}
Kostenlos starten →