EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access

Large Language Model–Based Assessment of Clinical Reasoning Documentation in the Electronic Health Record Across Two Institutions: Development and Validation Study

Verity Schaye iD, David DiTullio iD, Benedict Vincent Guzman iD, Scott Vennemeyer iD, Hanniel Shih iD, Ilan Reinstein iD, Danielle E Weber iD, Abbie Goodman iD, Danny T Y Wu iD, Daniel J Sartori iD, Sally A Santen iD, Larry Gruppen iD und 2 weitere

Journal of Medical Internet Research · 21.03.2025 · 27 · e67967

DOI: 10.2196/67967 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 21.03.2025. Verfasst von Verity Schaye und 13 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medicine, NYU Grossman School of Medicine, New York, NY, United States. Hauptthemen laut MEDLINE: Electronic Health Records, Documentation, Clinical Reasoning. Die Arbeit wurde bisher 21-mal zitiert (Crossref) und nennt 48 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Clinical reasoning (CR) is an essential skill; yet, physicians often receive limited feedback. Artificial intelligence holds promise to fill this gap.

Objective

We report the development of named entity recognition (NER), logic-based and large language model (LLM)–based assessments of CR documentation in the electronic health record across 2 institutions (New York University Grossman School of Medicine [NYU] and University of Cincinnati College of Medicine [UC]).

Methods

The note corpus consisted of internal medicine resident admission notes (retrospective set: July 2020-December 2021, n=700 NYU and 450 UC notes and prospective validation set: July 2023-December 2023, n=155 NYU and 92 UC notes). Clinicians rated CR documentation quality in each note using a previously validated tool (Revised-IDEA), on 3-point scales across 2 domains: differential diagnosis (D0, D1, and D2) and explanation of reasoning, (EA0, EA1, and EA2). At NYU, the retrospective set was annotated for NER for 5 entities (diagnosis, diagnostic category, prioritization of diagnosis language, data, and linkage terms). Models were developed using different artificial intelligence approaches, including NER, logic-based model: a large word vector model (scispaCy en_core_sci_lg) with model weights adjusted with backpropagation from annotations, developed at NYU with external validation at UC, NYUTron LLM: an NYU internal 110 million parameter LLM pretrained on 7.25 million clinical notes, only validated at NYU, and GatorTron LLM: an open source 345 million parameter LLM pretrained on 82 billion words of clinical text, fined tuned on NYU retrospective sets, then externally validated and further fine-tuned at UC. Model performance was assessed in the prospective sets with F1-scores for the NER, logic-based model and area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC) for the LLMs.

Results

At NYU, the NYUTron LLM performed best: the D0 and D2 models had AUROC/AUPRC 0.87/0.79 and 0.89/0.86, respectively. The D1, EA0, and EA1 models had insufficient performance for implementation (AUROC range 0.57-0.80, AUPRC range 0.33-0.63). For the D1 classification, the approach pivoted to a stepwise approach taking advantage of the more performant D0 and D2 models. For the EA model, the approach pivoted to a binary EA2 model (ie, EA2 vs not EA2) with excellent performance, AUROC/AUPRC 0.85/ 0.80. At UC, the NER, D-logic–based model was the best performing D model (F1-scores 0.80, 0.74, and 0.80 for D0, D1, D2, respectively. The GatorTron LLM performed best for EA2 scores AUROC/AUPRC 0.75/ 0.69.

Conclusions

This is the first multi-institutional study to apply LLMs for assessing CR documentation in the electronic health record. Such tools can enhance feedback on CR. Lessons learned by implementing these models at distinct institutions support the generalizability of this approach.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Electronic Health RecordsDocumentationClinical ReasoningHumansRetrospective StudiesArtificial IntelligenceNatural Language ProcessingLarge Language Models

Stichwörter der Autor:innen

artificial intelligenceassessmentclinical reasoningdocumentationelectronic health recordfeedbacklarge language models

Autor:innen und Institutionen

  • Department of Medicine, NYU Grossman School of Medicine, New York, NY, United States.
  • Institute for Innovations in Medical Education, NYU Grossman School of Medicine, New York, NY, United States.
  • Division of Applied AI Technologies, NYU Langone Health, New York, NY, United States.
  • Department of Biostatistics, Health informatics, and Data Sciences, University of Cincinnati College of Medicine, Cincinnati, OH, United States.
  • Division of Hospital Medicine, Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, OH, United States.
  • Division of Hospital Medicine, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, OH, United States.

ORCID-Profile: Verity Schaye, David DiTullio, Benedict Vincent Guzman, Scott Vennemeyer, Hanniel Shih, Ilan Reinstein, Danielle E Weber, Abbie Goodman, Danny T Y Wu, Daniel J Sartori, Sally A Santen, Larry Gruppen, Yindalon Aphinyanaphongs, Jesse Burk-Rafel

Zitieren

APA

Schaye, V., DiTullio, D., Guzman, B. V., Vennemeyer, S., Shih, H., Reinstein, I., Weber, D. E., Goodman, A., Wu, D. T. Y., Sartori, D. J., Santen, S. A., Gruppen, L., Aphinyanaphongs, Y., & Burk-Rafel, J. (2025). Large Language Model–Based Assessment of Clinical Reasoning Documentation in the Electronic Health Record Across Two Institutions: Development and Validation Study. Journal of Medical Internet Research, 27, e67967. https://doi.org/10.2196/67967

Vancouver

Schaye V, DiTullio D, Guzman BV, Vennemeyer S, Shih H, Reinstein I, et al. Large Language Model–Based Assessment of Clinical Reasoning Documentation in the Electronic Health Record Across Two Institutions: Development and Validation Study. Journal of Medical Internet Research. 2025;27:e67967. doi:10.2196/67967

BibTeX
@article{Schaye2025,
  author = {Schaye, Verity and DiTullio, David and Guzman, Benedict Vincent and Vennemeyer, Scott and Shih, Hanniel and Reinstein, Ilan and Weber, Danielle E and Goodman, Abbie and Wu, Danny T Y and Sartori, Daniel J and Santen, Sally A and Gruppen, Larry and Aphinyanaphongs, Yindalon and Burk-Rafel, Jesse},
  title = {Large Language Model–Based Assessment of Clinical Reasoning Documentation in the Electronic Health Record Across Two Institutions: Development and Validation Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e67967},
  doi = {10.2196/67967}
}
Kostenlos starten →