EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Development of a benchmarking dataset for symptom detection using large language models

Joshua Davis iD, Brigitte N Durieux iD, Chloe Van Dongen, Kate Sciacca, Juan Manuel Gutierrez, Charlotta Lindvall

JAMIA Open · 07.07.2026 · 9 (4)

DOI: 10.1093/jamiaopen/ooag134 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 10.07.2026. Verfasst von Joshua Davis und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Supportive Oncology, Dana-Farber Cancer Institute, Boston, MA, 02215, United States. Die Arbeit nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

To develop a pipeline for evaluating large language models (LLMs) on the task of capturing symptoms from clinical encounters. Materials and Methods We created a gold standard dataset of symptom annotations from simulated doctor-patient encounter excerpts (264 encounters; 16 symptoms; double-coded and adjudicated). Nine different LLMs from 4 vendors (OpenAI, Meta, DeepSeek, Moonshot AI) were used as examples to test our evaluation pipeline; outputs were assessed for correct structure and symptom information.

Results

Of 3085 excerpts, 2087 (68%) contained symptoms. Pain, cough, and shortness of breath were most common; LLMs achieved F1 scores ranging 0.66-0.88 for these symptoms with minimal prompt engineering. Of tested models, GPT-4.1 demonstrated the best overall performance.

Discussion

Our evaluation pipeline and benchmarking dataset are publicly available and applicable to various LLMs, including open-source models.

Conclusion

This work supports the development and optimization of models that seek to improve patient symptom understanding.

Schlagwörter

Stichwörter der Autor:innen

artificial intelligencecomputing methodologiesnatural language processingsigns and symptomssymptom assessment

Autor:innen und Institutionen

  • Department of Supportive Oncology, Dana-Farber Cancer Institute, Boston, MA, 02215, United States.
  • Albany Medical College, Albany, NY, 12208, United States.
  • Department of Informatics & Analytics, Dana-Farber Cancer Institute, Boston, MA, 02215, United States.
  • Harvard Medical School, Harvard University, Boston, MA, 02115, United States.

ORCID-Profile: Joshua Davis, Brigitte N Durieux

Zitieren

APA

Davis, J., Durieux, B. N., Van Dongen, C., Sciacca, K., Gutierrez, J. M., & Lindvall, C. (2026). Development of a benchmarking dataset for symptom detection using large language models. JAMIA Open, 9(4). https://doi.org/10.1093/jamiaopen/ooag134

Vancouver

Davis J, Durieux BN, Van Dongen C, Sciacca K, Gutierrez JM, Lindvall C. Development of a benchmarking dataset for symptom detection using large language models. JAMIA Open. 2026;9(4). doi:10.1093/jamiaopen/ooag134

BibTeX
@article{Davis2026,
  author = {Davis, Joshua and Durieux, Brigitte N and Van Dongen, Chloe and Sciacca, Kate and Gutierrez, Juan Manuel and Lindvall, Charlotta},
  title = {Development of a benchmarking dataset for symptom detection using large language models},
  journal = {JAMIA Open},
  year = {2026},
  volume = {9},
  number = {4},
  doi = {10.1093/jamiaopen/ooag134}
}
Kostenlos starten →