EUVIMEDEUVIMED entdecken
Open Access

Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study

Theresa Isabelle Wilhelm iD, Jonas Roos iD, Robert Kaczmarczyk iD

Journal of Medical Internet Research · 30.10.2023 · 25 · e49324

DOI: 10.2196/49324 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 30.10.2023. Verfasst von Theresa Isabelle Wilhelm und 2 weiteren Autor:innen. Beteiligte Einrichtung: Eye Center, Medical Center, Faculty of Medicine, University of Freiburg, Freiburg, Germany. Hauptthemen laut MEDLINE: Artificial Intelligence, Medicine. Die Arbeit wurde bisher 145-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

As advancements in artificial intelligence (AI) continue, large language models (LLMs) have emerged as promising tools for generating medical information. Their rapid adaptation and potential benefits in health care require rigorous assessment in terms of the quality, accuracy, and safety of the generated information across diverse medical specialties.

Objective

This study aimed to evaluate the performance of 4 prominent LLMs, namely, Claude-instant-v1.0, GPT-3.5-Turbo, Command-xlarge-nightly, and Bloomz, in generating medical content spanning the clinical specialties of ophthalmology, orthopedics, and dermatology.

Methods

Three domain-specific physicians evaluated the AI-generated therapeutic recommendations for a diverse set of 60 diseases. The evaluation criteria involved the mDISCERN score, correctness, and potential harmfulness of the recommendations. ANOVA and pairwise t tests were used to explore discrepancies in content quality and safety across models and specialties. Additionally, using the capabilities of OpenAI’s most advanced model, GPT-4, an automated evaluation of each model’s responses to the diseases was performed using the same criteria and compared to the physicians’ assessments through Pearson correlation analysis.

Results

Claude-instant-v1.0 emerged with the highest mean mDISCERN score (3.35, 95% CI 3.23-3.46). In contrast, Bloomz lagged with the lowest score (1.07, 95% CI 1.03-1.10). Our analysis revealed significant differences among the models in terms of quality (P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceMedicineHumansQuality of LifeReproducibility of ResultsLanguage

Stichwörter der Autor:innen

ChatGPTLLMaccuracyartificial intelligencechatbotchatbotsdermatologyhealth informationlarge language modelsmedical advicemedical informationophthalmologyorthopedicorthopedicsquality

Autor:innen und Institutionen

  • Eye Center, Medical Center, Faculty of Medicine, University of Freiburg, Freiburg, Germany.
  • Medical Graduate Center, School of Medicine, Technical University of Munich, Munich, Germany.
  • Department of Orthopedics and Trauma Surgery, University Hospital of Bonn, Bonn, Germany.
  • Department of Dermatology and Allergy, School of Medicine, Technical University of Munich, Munich, Germany.
  • Division of Dermatology and Venerology, Department of Medicine Solna, Karolinska Institutet, Solna, Sweden.

ORCID-Profile: Theresa Isabelle Wilhelm, Jonas Roos, Robert Kaczmarczyk

Zitieren

APA

Wilhelm, T. I., Roos, J., & Kaczmarczyk, R. (2023). Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study. Journal of Medical Internet Research, 25, e49324. https://doi.org/10.2196/49324

Vancouver

Wilhelm TI, Roos J, Kaczmarczyk R. Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study. Journal of Medical Internet Research. 2023;25:e49324. doi:10.2196/49324

BibTeX
@article{Wilhelm2023,
  author = {Wilhelm, Theresa Isabelle and Roos, Jonas and Kaczmarczyk, Robert},
  title = {Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study},
  journal = {Journal of Medical Internet Research},
  year = {2023},
  volume = {25},
  pages = {e49324},
  doi = {10.2196/49324}
}
Kostenlos starten →