Accuracy of Large Language Models in Generating Rare Disease Differential Diagnosis Using Key Clinical Features
Studies in Health Technology and Informatics · 2025
DOI: 10.3233/shti251000 ↗Kurz gefasst
Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Cathy Shyr und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA. Hauptthemen laut MEDLINE: Rare Diseases, Natural Language Processing, Diagnosis, Computer-Assisted. Die Arbeit wurde bisher 3-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: NLM NIH HHS, NICHD NIH HHS, NINDS NIH HHS.
Abstract (Originalsprache)
Generating differential diagnoses for rare disease patients can be time intensive and highly dependent on the background and training of the evaluating physicians. Large language models (LLMs) have the potential to complement this process by automatically generating differentials to support physicians, but their performance in real-world patient populations remains underexplored. To this end, we assessed the diagnostic accuracy of ChatGPT-4o, Llama 3.1-8B-Instruct, and Exomiser in 424 rare disease patients at the Undiagnosed Diseases Network. ChatGPT-4o had the highest differential diagnostic accuracy (22.4% [95% CI: 18.4, 26.4]), outperforming Exomiser (13.9% [10.6, 17.2]; p < 0.001) and Llama 3.1-8B-Instruct (11.6% [8.5, 14.6]; p < 0.001). Adjusting for other factors, age at symptom onset was a significant predictor of ChatGPT-4o’s diagnostic accuracy with the model performing better in patients with later symptom onset, potentially due to more distinct phenotypic presentations in older individuals. The combined accuracy of ChatGPT-4o and Exomiser was 30% [25.6, 34.3] and higher than that of either model alone (p < 0.01). This improvement highlights the potential of combining LLMs and bioinformatic models to generate differential diagnoses for rare diseases.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA.
- Department of Pediatrics, Vanderbilt University Medical Center, Nashville, TN, USA.
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN, USA.
- Department of Biomedical Informatics and Data Science, Yale School of Medicine, New Haven, CT, USA.
Förderung
- NLM NIH HHS
- NICHD NIH HHS
- NINDS NIH HHS
Zitieren
APA
Shyr, C., Tinker, R. J., Harris, P. A., Cheng, A. C., Byram, K. W., Bastarache, L., Peterson, J. F., Hamid, R., Xu, H., & Cassini, T. A. (2025). Accuracy of Large Language Models in Generating Rare Disease Differential Diagnosis Using Key Clinical Features. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti251000
Vancouver
Shyr C, Tinker RJ, Harris PA, Cheng AC, Byram KW, Bastarache L, et al. Accuracy of Large Language Models in Generating Rare Disease Differential Diagnosis Using Key Clinical Features. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti251000
BibTeX
@article{Shyr2025,
author = {Shyr, Cathy and Tinker, Rory J and Harris, Paul A and Cheng, Alex C and Byram, Kevin W and Bastarache, Lisa and Peterson, Josh F and Hamid, Rizwan and Xu, Hua and Cassini, Thomas A},
title = {Accuracy of Large Language Models in Generating Rare Disease Differential Diagnosis Using Key Clinical Features},
journal = {Studies in Health Technology and Informatics},
year = {2025},
doi = {10.3233/shti251000}
}
EUVIMED