EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Accuracy of Large Language Models in Generating Rare Disease Differential Diagnosis Using Key Clinical Features

Cathy Shyr, Rory J Tinker, Paul A Harris, Alex C Cheng, Kevin W Byram, Lisa Bastarache, Josh F Peterson, Rizwan Hamid, Hua Xu, Thomas A Cassini

Studies in Health Technology and Informatics · 2025

DOI: 10.3233/shti251000 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Cathy Shyr und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA. Hauptthemen laut MEDLINE: Rare Diseases, Natural Language Processing, Diagnosis, Computer-Assisted. Die Arbeit wurde bisher 3-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: NLM NIH HHS, NICHD NIH HHS, NINDS NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Generating differential diagnoses for rare disease patients can be time intensive and highly dependent on the background and training of the evaluating physicians. Large language models (LLMs) have the potential to complement this process by automatically generating differentials to support physicians, but their performance in real-world patient populations remains underexplored. To this end, we assessed the diagnostic accuracy of ChatGPT-4o, Llama 3.1-8B-Instruct, and Exomiser in 424 rare disease patients at the Undiagnosed Diseases Network. ChatGPT-4o had the highest differential diagnostic accuracy (22.4% [95% CI: 18.4, 26.4]), outperforming Exomiser (13.9% [10.6, 17.2]; p < 0.001) and Llama 3.1-8B-Instruct (11.6% [8.5, 14.6]; p < 0.001). Adjusting for other factors, age at symptom onset was a significant predictor of ChatGPT-4o’s diagnostic accuracy with the model performing better in patients with later symptom onset, potentially due to more distinct phenotypic presentations in older individuals. The combined accuracy of ChatGPT-4o and Exomiser was 30% [25.6, 34.3] and higher than that of either model alone (p < 0.01). This improvement highlights the potential of combining LLMs and bioinformatic models to generate differential diagnoses for rare diseases.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Rare DiseasesNatural Language ProcessingDiagnosis, Computer-AssistedDiagnosis, DifferentialHumansMaleFemaleLarge Language Models

Stichwörter der Autor:innen

ExomiserRare diseasedifferential diagnosislarge language models

Autor:innen und Institutionen

  • Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA.
  • Department of Pediatrics, Vanderbilt University Medical Center, Nashville, TN, USA.
  • Department of Medicine, Vanderbilt University Medical Center, Nashville, TN, USA.
  • Department of Biomedical Informatics and Data Science, Yale School of Medicine, New Haven, CT, USA.

Förderung

  • NLM NIH HHS
  • NICHD NIH HHS
  • NINDS NIH HHS

Zitieren

APA

Shyr, C., Tinker, R. J., Harris, P. A., Cheng, A. C., Byram, K. W., Bastarache, L., Peterson, J. F., Hamid, R., Xu, H., & Cassini, T. A. (2025). Accuracy of Large Language Models in Generating Rare Disease Differential Diagnosis Using Key Clinical Features. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti251000

Vancouver

Shyr C, Tinker RJ, Harris PA, Cheng AC, Byram KW, Bastarache L, et al. Accuracy of Large Language Models in Generating Rare Disease Differential Diagnosis Using Key Clinical Features. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti251000

BibTeX
@article{Shyr2025,
  author = {Shyr, Cathy and Tinker, Rory J and Harris, Paul A and Cheng, Alex C and Byram, Kevin W and Bastarache, Lisa and Peterson, Josh F and Hamid, Rizwan and Xu, Hua and Cassini, Thomas A},
  title = {Accuracy of Large Language Models in Generating Rare Disease Differential Diagnosis Using Key Clinical Features},
  journal = {Studies in Health Technology and Informatics},
  year = {2025},
  doi = {10.3233/shti251000}
}
Kostenlos starten →