EUVIMEDOpdag EUVIMED
Open access

Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study

Theresa Isabelle Wilhelm iD, Jonas Roos iD, Robert Kaczmarczyk iD

Journal of Medical Internet Research · 2023-10-30 · 25 · e49324

DOI: 10.2196/49324 ↗

At a glance

Journal article in Journal of Medical Internet Research, published online on 2023-10-30. Written by Theresa Isabelle Wilhelm and 2 co-authors. Institution: Eye Center, Medical Center, Faculty of Medicine, University of Freiburg, Freiburg, Germany. Major MEDLINE topics: Artificial Intelligence, Medicine. Cited 145 times so far (Crossref); 37 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

As advancements in artificial intelligence (AI) continue, large language models (LLMs) have emerged as promising tools for generating medical information. Their rapid adaptation and potential benefits in health care require rigorous assessment in terms of the quality, accuracy, and safety of the generated information across diverse medical specialties.

Objective

This study aimed to evaluate the performance of 4 prominent LLMs, namely, Claude-instant-v1.0, GPT-3.5-Turbo, Command-xlarge-nightly, and Bloomz, in generating medical content spanning the clinical specialties of ophthalmology, orthopedics, and dermatology.

Methods

Three domain-specific physicians evaluated the AI-generated therapeutic recommendations for a diverse set of 60 diseases. The evaluation criteria involved the mDISCERN score, correctness, and potential harmfulness of the recommendations. ANOVA and pairwise t tests were used to explore discrepancies in content quality and safety across models and specialties. Additionally, using the capabilities of OpenAI’s most advanced model, GPT-4, an automated evaluation of each model’s responses to the diseases was performed using the same criteria and compared to the physicians’ assessments through Pearson correlation analysis.

Results

Claude-instant-v1.0 emerged with the highest mean mDISCERN score (3.35, 95% CI 3.23-3.46). In contrast, Bloomz lagged with the lowest score (1.07, 95% CI 1.03-1.10). Our analysis revealed significant differences among the models in terms of quality (P

Subjects

MeSH terms (MEDLINE)

Artificial IntelligenceMedicineHumansQuality of LifeReproducibility of ResultsLanguage

Author keywords

ChatGPTLLMaccuracyartificial intelligencechatbotchatbotsdermatologyhealth informationlarge language modelsmedical advicemedical informationophthalmologyorthopedicorthopedicsquality

Authors and institutions

  • Eye Center, Medical Center, Faculty of Medicine, University of Freiburg, Freiburg, Germany.
  • Medical Graduate Center, School of Medicine, Technical University of Munich, Munich, Germany.
  • Department of Orthopedics and Trauma Surgery, University Hospital of Bonn, Bonn, Germany.
  • Department of Dermatology and Allergy, School of Medicine, Technical University of Munich, Munich, Germany.
  • Division of Dermatology and Venerology, Department of Medicine Solna, Karolinska Institutet, Solna, Sweden.

ORCID profiles: Theresa Isabelle Wilhelm, Jonas Roos, Robert Kaczmarczyk

Cite

APA

Wilhelm, T. I., Roos, J., & Kaczmarczyk, R. (2023). Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study. Journal of Medical Internet Research, 25, e49324. https://doi.org/10.2196/49324

Vancouver

Wilhelm TI, Roos J, Kaczmarczyk R. Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study. Journal of Medical Internet Research. 2023;25:e49324. doi:10.2196/49324

BibTeX
@article{Wilhelm2023,
  author = {Wilhelm, Theresa Isabelle and Roos, Jonas and Kaczmarczyk, Robert},
  title = {Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study},
  journal = {Journal of Medical Internet Research},
  year = {2023},
  volume = {25},
  pages = {e49324},
  doi = {10.2196/49324}
}
Start for free →