EUVIMEDTutustu EUVIMEDiin
BeobachtungsstudieOpen access

The Performance of Large Language Models in Extracting Intestinal Symptoms From Electronic Health Records: Retrospective Observational Study

Xinyue Zhang, Quanyu Wang, Beibei Liu, Xinyi Sang, Sheng Wei

Journal of Medical Internet Research · 2026-08-26 · 28 · e98580-e98580

DOI: 10.2196/98580 ↗

At a glance

Beobachtungsstudie in Journal of Medical Internet Research, published online on 2026-08-26. Written by Xinyue Zhang and 4 co-authors. Institution: School of Public Health and Emergency Management, Southern University of Science and Technology, 1088 Xueyuan Avenue, Shenzhen, Guangdong, 518055, Ch…. Major MEDLINE topics: Electronic Health Records, Intestinal Diseases. 43 references. The full text is freely available in PubMed Central.

TiivistelmäSubjectsBibliografiset tiedotCiteRelated studies

Abstract (original language)

Background

Unstructured electronic health records (EHRs) hinder the monitoring of intestinal infections. Large language models (LLMs) enable automated symptom extraction. However, their clinical validation is limited by a lack of systematic multimodel comparisons, unclear prompting strategies, and the privacy risks of cloud-based models (eg, data leakage and cross-border data transfer).

Objective

This study aimed to systematically evaluate the performance of locally deployed open-source LLMs across 4 model families in extracting intestinal symptoms from unstructured EHR chief complaints under different prompting strategies.

Methods

From a citywide health care information platform in Wuhan, China, we randomly selected 1000 chief complaints from outpatient records of intestinal clinics, infectious disease departments, pediatrics, and fever clinics. Six symptoms related to intestinal infectious diseases—diarrhea/bloody/mucoid stools, vomiting, abdominal pain, fever, nausea, and rash—were manually annotated as a gold-standard dataset. Twelve locally deployed open-source LLMs across 4 families, namely, Gemma3 (1b, 4b, 12b), Qwen3 (1.7b, 8b, 14b), DeepSeek-R1 (1.5b, 7b, 14b), and Llama (Llama2-Chinese 7b, 13b; Llama3.1 8b), were evaluated on the symptom extraction task using the gold-standard dataset. Three prompting strategies (no-role, zero-shot, and few-shot) were tested. Performance metrics included accuracy, precision, recall, F 1 -score, specificity, balanced accuracy, and inference time. Statistical comparisons used Friedman tests for global differences, followed by Wilcoxon signed-rank and Mann-Whitney U tests with Bonferroni and false discovery rate corrections for pairwise comparisons.

Results

Among the 4 families, Qwen3 models showed higher F 1 -scores and balanced accuracy, with Qwen3-1.7b achieving a macroaveraged F 1 -score of 0.85 under zero-shot prompting and Qwen3-8b reaching 0.89 under no-role prompting, while Gemma3 demonstrated robust performance at small to medium scales. Symptom-wise, models agreed more on frequent symptoms such as diarrhea and fever, whereas greater variability was observed for rarer symptoms like rash and nausea. The effect of prompting strategy varied across models, with no single strategy consistently outperforming the others. Although some pairwise differences reached statistical significance ( P

Subjects

MeSH terms (MEDLINE)

Electronic Health RecordsIntestinal DiseasesLarge Language ModelsRetrospective StudiesHumansChina

Author keywords

electronic health recordsintestinal infectious diseaselarge language modelssurveillancesymptom extraction

Authors and institutions

  • School of Public Health and Emergency Management, Southern University of Science and Technology, 1088 Xueyuan Avenue, Shenzhen, Guangdong, 518055, China, 86 755-88011926, 86 755-88011926.
  • Department of Epidemiology and Biostatistics, Tongji Medical College, School of Public Health, Huazhong University of Science and Technology, Wuhan, Hubei, China.

Cite

APA

Zhang, X., Wang, Q., Liu, B., Sang, X., & Wei, S. (2026). The Performance of Large Language Models in Extracting Intestinal Symptoms From Electronic Health Records: Retrospective Observational Study. Journal of Medical Internet Research, 28, e98580-e98580. https://doi.org/10.2196/98580

Vancouver

Zhang X, Wang Q, Liu B, Sang X, Wei S. The Performance of Large Language Models in Extracting Intestinal Symptoms From Electronic Health Records: Retrospective Observational Study. Journal of Medical Internet Research. 2026;28:e98580-e98580. doi:10.2196/98580

BibTeX
@article{Zhang2026,
  author = {Zhang, Xinyue and Wang, Quanyu and Liu, Beibei and Sang, Xinyi and Wei, Sheng},
  title = {The Performance of Large Language Models in Extracting Intestinal Symptoms From Electronic Health Records: Retrospective Observational Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e98580--e98580},
  doi = {10.2196/98580}
}
Start for free →