Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study
Journal of Medical Internet Research · 2023-10-30 · 25 · e49324
DOI: 10.2196/49324 ↗At a glance
Journal article in Journal of Medical Internet Research, published online on 2023-10-30. Written by Theresa Isabelle Wilhelm and 2 co-authors. Institution: Eye Center, Medical Center, Faculty of Medicine, University of Freiburg, Freiburg, Germany. Major MEDLINE topics: Artificial Intelligence, Medicine. Cited 145 times so far (Crossref); 37 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
As advancements in artificial intelligence (AI) continue, large language models (LLMs) have emerged as promising tools for generating medical information. Their rapid adaptation and potential benefits in health care require rigorous assessment in terms of the quality, accuracy, and safety of the generated information across diverse medical specialties.
Objective
This study aimed to evaluate the performance of 4 prominent LLMs, namely, Claude-instant-v1.0, GPT-3.5-Turbo, Command-xlarge-nightly, and Bloomz, in generating medical content spanning the clinical specialties of ophthalmology, orthopedics, and dermatology.
Methods
Three domain-specific physicians evaluated the AI-generated therapeutic recommendations for a diverse set of 60 diseases. The evaluation criteria involved the mDISCERN score, correctness, and potential harmfulness of the recommendations. ANOVA and pairwise t tests were used to explore discrepancies in content quality and safety across models and specialties. Additionally, using the capabilities of OpenAI’s most advanced model, GPT-4, an automated evaluation of each model’s responses to the diseases was performed using the same criteria and compared to the physicians’ assessments through Pearson correlation analysis.
Results
Claude-instant-v1.0 emerged with the highest mean mDISCERN score (3.35, 95% CI 3.23-3.46). In contrast, Bloomz lagged with the lowest score (1.07, 95% CI 1.03-1.10). Our analysis revealed significant differences among the models in terms of quality (P
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Eye Center, Medical Center, Faculty of Medicine, University of Freiburg, Freiburg, Germany.
- Medical Graduate Center, School of Medicine, Technical University of Munich, Munich, Germany.
- Department of Orthopedics and Trauma Surgery, University Hospital of Bonn, Bonn, Germany.
- Department of Dermatology and Allergy, School of Medicine, Technical University of Munich, Munich, Germany.
- Division of Dermatology and Venerology, Department of Medicine Solna, Karolinska Institutet, Solna, Sweden.
ORCID profiles: Theresa Isabelle Wilhelm, Jonas Roos, Robert Kaczmarczyk
Cite
APA
Wilhelm, T. I., Roos, J., & Kaczmarczyk, R. (2023). Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study. Journal of Medical Internet Research, 25, e49324. https://doi.org/10.2196/49324
Vancouver
Wilhelm TI, Roos J, Kaczmarczyk R. Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study. Journal of Medical Internet Research. 2023;25:e49324. doi:10.2196/49324
BibTeX
@article{Wilhelm2023,
author = {Wilhelm, Theresa Isabelle and Roos, Jonas and Kaczmarczyk, Robert},
title = {Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study},
journal = {Journal of Medical Internet Research},
year = {2023},
volume = {25},
pages = {e49324},
doi = {10.2196/49324}
}
EUVIMED