EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access · CC BY 4.0

Vignette-based comparative analysis of ChatGPT and specialist treatment decisions for rheumatic patients: results of the Rheum2Guide study

Hannah Labinsky iD, Lea-Kristin Nagler iD, Martin Krusche iD, Sebastian Griewing iD, Peer Aries iD, Anja Kroiß iD, Patrick-Pascal Strunz iD, Sebastian Kuhn iD, Marc Schmalzing iD, Michael Gernert iD, Johannes Knitza iD

Rheumatology International · 2024

DOI: 10.1007/s00296-024-05675-5 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Rheumatology International, 2024. Verfasst von Hannah Labinsky und 10 weiteren Autor:innen. Beteiligte Einrichtung: Department of Internal Medicine 2, Rheumatology/Clinical Immunology, University Hospital Würzburg, Oberdürrbacher Straße 6, 97080, Würzburg, Germany. Hauptthemen laut MEDLINE: Rheumatic Diseases, Clinical Decision-Making. Die Arbeit wurde bisher 27-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: GlaxoSmithKline Biologicals, Philipps-Universität Marburg.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The complex nature of rheumatic diseases poses considerable challenges for clinicians when developing individualized treatment plans. Large language models (LLMs) such as ChatGPT could enable treatment decision support.

Objective

To compare treatment plans generated by ChatGPT-3.5 and GPT-4 to those of a clinical rheumatology board (RB). Design/methods Fictional patient vignettes were created and GPT-3.5, GPT-4, and the RB were queried to provide respective first- and second-line treatment plans with underlying justifications. Four rheumatologists from different centers, blinded to the origin of treatment plans, selected the overall preferred treatment concept and assessed treatment plans’ safety, EULAR guideline adherence, medical adequacy, overall quality, justification of the treatment plans and their completeness as well as patient vignette difficulty using a 5-point Likert scale.

Results

20 fictional vignettes covering various rheumatic diseases and varying difficulty levels were assembled and a total of 160 ratings were assessed. In 68.8% (110/160) of cases, raters preferred the RB’s treatment plans over those generated by GPT-4 (16.3%; 26/160) and GPT-3.5 (15.0%; 24/160). GPT-4’s plans were chosen more frequently for first-line treatments compared to GPT-3.5. No significant safety differences were observed between RB and GPT-4’s first-line treatment plans. Rheumatologists’ plans received significantly higher ratings in guideline adherence, medical appropriateness, completeness and overall quality. Ratings did not correlate with the vignette difficulty. LLM-generated plans were notably longer and more detailed.

Conclusion

GPT-4 and GPT-3.5 generated safe, high-quality treatment plans for rheumatic diseases, demonstrating promise in clinical decision support. Future research should investigate detailed standardized prompts and the impact of LLM usage on clinical decisions.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Rheumatic DiseasesClinical Decision-MakingHumansDecision Support TechniquesGuideline AdherenceRheumatologyFemaleMaleRheumatologistsPatient Care PlanningPractice Guidelines as Topic

Stichwörter der Autor:innen

Artificial intelligenceChatGPTClinical decision support system (CDSS)Large language model

Autor:innen und Institutionen

  • Department of Internal Medicine 2, Rheumatology/Clinical Immunology, University Hospital Würzburg, Oberdürrbacher Straße 6, 97080, Würzburg, Germany.
  • Division of Rheumatology and Systemic Inflammatory Diseases, III. Department of Medicine, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
  • Institute for Digital Medicine, University Hospital Giessen-Marburg, Philipps University, Baldingerstrasse, Marburg, Germany.
  • Stanford Center for Biomedical Informatics Research, Stanford University School of Medicine, Palo Alto, CA, USA.
  • Department of Rheumatology, Immunologikum, Hamburg, Germany.
  • Institute for Digital Medicine, University Hospital Giessen-Marburg, Philipps University, Baldingerstrasse, Marburg, Germany. knitza@uni-marburg.de.

ORCID-Profile: Hannah Labinsky, Lea-Kristin Nagler, Martin Krusche, Sebastian Griewing, Peer Aries, Anja Kroiß, Patrick-Pascal Strunz, Sebastian Kuhn, Marc Schmalzing, Michael Gernert, Johannes Knitza

Förderung

  • GlaxoSmithKline Biologicals
  • Philipps-Universität Marburg

Zitieren

APA

Labinsky, H., Nagler, L. K., Krusche, M., Griewing, S., Aries, P., Kroiß, A., Strunz, P. P., Kuhn, S., Schmalzing, M., Gernert, M., & Knitza, J. (2024). Vignette-based comparative analysis of ChatGPT and specialist treatment decisions for rheumatic patients: results of the Rheum2Guide study. Rheumatology International. https://doi.org/10.1007/s00296-024-05675-5

Vancouver

Labinsky H, Nagler LK, Krusche M, Griewing S, Aries P, Kroiß A, et al. Vignette-based comparative analysis of ChatGPT and specialist treatment decisions for rheumatic patients: results of the Rheum2Guide study. Rheumatology International. 2024. doi:10.1007/s00296-024-05675-5

BibTeX
@article{Labinsky2024,
  author = {Labinsky, Hannah and Nagler, Lea-Kristin and Krusche, Martin and Griewing, Sebastian and Aries, Peer and Kroiß, Anja and Strunz, Patrick-Pascal and Kuhn, Sebastian and Schmalzing, Marc and Gernert, Michael and Knitza, Johannes},
  title = {Vignette-based comparative analysis of ChatGPT and specialist treatment decisions for rheumatic patients: results of the Rheum2Guide study},
  journal = {Rheumatology International},
  year = {2024},
  doi = {10.1007/s00296-024-05675-5}
}
Kostenlos starten →