EUVIMEDEUVIMED entdecken
Open Access

Classifying Unstructured Text in Electronic Health Records for Mental Health Prediction Models: Large Language Model Evaluation Study

Nicholas C Cardamone iD, Mark Olfson iD, Timothy Schmutte iD, Lyle Ungar iD, Tony Liu iD, Sara W Cullen iD, Nathaniel J Williams iD, Steven C Marcus iD

JMIR Medical Informatics · 21.01.2025 · 13 · e65454-e65454

DOI: 10.2196/65454 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Medical Informatics, online erschienen am 21.01.2025. Verfasst von Nicholas C Cardamone und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States. Hauptthemen laut MEDLINE: Electronic Health Records, Mental Disorders, Natural Language Processing, Mental Health. Die Arbeit wurde bisher 37-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIMH NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Prediction models have demonstrated a range of applications across medicine, including using electronic health record (EHR) data to identify hospital readmission and mortality risk. Large language models (LLMs) can transform unstructured EHR text into structured features, which can then be integrated into statistical prediction models, ensuring that the results are both clinically meaningful and interpretable.

Objective

This study aims to compare the classification decisions made by clinical experts with those generated by a state-of-the-art LLM, using terms extracted from a large EHR data set of individuals with mental health disorders seen in emergency departments (EDs).

Methods

Using a dataset from the EHR systems of more than 50 health care provider organizations in the United States from 2016 to 2021, we extracted all clinical terms that appeared in at least 1000 records of individuals admitted to the ED for a mental health–related problem from a source population of over 6 million ED episodes. Two experienced mental health clinicians (one medically trained psychiatrist and one clinical psychologist) reached consensus on the classification of EHR terms and diagnostic codes into categories. We evaluated an LLM’s agreement with clinical judgment across three classification tasks as follows: (1) classify terms into “mental health” or “physical health”, (2) classify mental health terms into 1 of 42 prespecified categories, and (3) classify physical health terms into 1 of 19 prespecified broad categories.

Results

There was high agreement between the LLM and clinical experts when categorizing 4553 terms as “mental health” or “physical health” (κ=0.77, 95% CI 0.75-0.80). However, there was still considerable variability in LLM-clinician agreement on the classification of mental health terms (κ=0.62, 95% CI 0.59‐0.66) and physical health terms (κ=0.69, 95% CI 0.67‐0.70).

Conclusions

The LLM displayed high agreement with clinical experts when classifying EHR terms into certain mental health or physical health term categories. However, agreement with clinical experts varied considerably within both sets of mental and physical health term categories. Importantly, the use of LLMs presents an alternative to manual human coding, presenting great potential to create interpretable features for prediction models.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Electronic Health RecordsMental DisordersNatural Language ProcessingMental HealthHumansEmergency Service, HospitalUnited StatesLarge Language Models

Stichwörter der Autor:innen

AIChatGPTEHREHR systemLLMMLNLPartificial intelligencedatasetelectronic health recordemergency departmenthealth informaticslarge language modelmachine learningmental health

Autor:innen und Institutionen

  • Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States.
  • Department of Psychiatry, the New York State Psychiatric Institute, New York, NY, United States.
  • Department of Psychiatry, Yale School of Medicine, New Haven, CT, United States.
  • Computer and Information Science, University of Pennsylvania, Philadelphia, PA, United States.
  • School of Social Policy & Practice, University of Pennsylvania, Philadelphia, PA, United States.
  • School of Social Work, Boise State University, Boise, ID, United States.

ORCID-Profile: Nicholas C Cardamone, Mark Olfson, Timothy Schmutte, Lyle Ungar, Tony Liu, Sara W Cullen, Nathaniel J Williams, Steven C Marcus

Förderung

  • NIMH NIH HHS

Zitieren

APA

Cardamone, N. C., Olfson, M., Schmutte, T., Ungar, L., Liu, T., Cullen, S. W., Williams, N. J., & Marcus, S. C. (2025). Classifying Unstructured Text in Electronic Health Records for Mental Health Prediction Models: Large Language Model Evaluation Study. JMIR Medical Informatics, 13, e65454-e65454. https://doi.org/10.2196/65454

Vancouver

Cardamone NC, Olfson M, Schmutte T, Ungar L, Liu T, Cullen SW, et al. Classifying Unstructured Text in Electronic Health Records for Mental Health Prediction Models: Large Language Model Evaluation Study. JMIR Medical Informatics. 2025;13:e65454-e65454. doi:10.2196/65454

BibTeX
@article{Cardamone2025,
  author = {Cardamone, Nicholas C and Olfson, Mark and Schmutte, Timothy and Ungar, Lyle and Liu, Tony and Cullen, Sara W and Williams, Nathaniel J and Marcus, Steven C},
  title = {Classifying Unstructured Text in Electronic Health Records for Mental Health Prediction Models: Large Language Model Evaluation Study},
  journal = {JMIR Medical Informatics},
  year = {2025},
  volume = {13},
  pages = {e65454--e65454},
  doi = {10.2196/65454}
}
Kostenlos starten →