Development of a benchmarking dataset for symptom detection using large language models
JAMIA Open · 07.07.2026 · 9 (4)
DOI: 10.1093/jamiaopen/ooag134 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 10.07.2026. Verfasst von Joshua Davis und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Supportive Oncology, Dana-Farber Cancer Institute, Boston, MA, 02215, United States. Die Arbeit nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Objectives
To develop a pipeline for evaluating large language models (LLMs) on the task of capturing symptoms from clinical encounters. Materials and Methods We created a gold standard dataset of symptom annotations from simulated doctor-patient encounter excerpts (264 encounters; 16 symptoms; double-coded and adjudicated). Nine different LLMs from 4 vendors (OpenAI, Meta, DeepSeek, Moonshot AI) were used as examples to test our evaluation pipeline; outputs were assessed for correct structure and symptom information.
Results
Of 3085 excerpts, 2087 (68%) contained symptoms. Pain, cough, and shortness of breath were most common; LLMs achieved F1 scores ranging 0.66-0.88 for these symptoms with minimal prompt engineering. Of tested models, GPT-4.1 demonstrated the best overall performance.
Discussion
Our evaluation pipeline and benchmarking dataset are publicly available and applicable to various LLMs, including open-source models.
Conclusion
This work supports the development and optimization of models that seek to improve patient symptom understanding.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Supportive Oncology, Dana-Farber Cancer Institute, Boston, MA, 02215, United States.
- Albany Medical College, Albany, NY, 12208, United States.
- Department of Informatics & Analytics, Dana-Farber Cancer Institute, Boston, MA, 02215, United States.
- Harvard Medical School, Harvard University, Boston, MA, 02115, United States.
ORCID-Profile: Joshua Davis, Brigitte N Durieux
Zitieren
APA
Davis, J., Durieux, B. N., Van Dongen, C., Sciacca, K., Gutierrez, J. M., & Lindvall, C. (2026). Development of a benchmarking dataset for symptom detection using large language models. JAMIA Open, 9(4). https://doi.org/10.1093/jamiaopen/ooag134
Vancouver
Davis J, Durieux BN, Van Dongen C, Sciacca K, Gutierrez JM, Lindvall C. Development of a benchmarking dataset for symptom detection using large language models. JAMIA Open. 2026;9(4). doi:10.1093/jamiaopen/ooag134
BibTeX
@article{Davis2026,
author = {Davis, Joshua and Durieux, Brigitte N and Van Dongen, Chloe and Sciacca, Kate and Gutierrez, Juan Manuel and Lindvall, Charlotta},
title = {Development of a benchmarking dataset for symptom detection using large language models},
journal = {JAMIA Open},
year = {2026},
volume = {9},
number = {4},
doi = {10.1093/jamiaopen/ooag134}
}
EUVIMED