EUVIMEDEUVIMED entdecken
BeobachtungsstudieVergleichsstudieLeitlinieOpen Access · CC BY 4.0

A patient-derived benchmark for evaluating large language models in connective tissue diseases: blinded multi-stakeholder assessment and guideline comparison

Phillip Kremer, Johannes Knitza, Anna Reichard, Marie Therese Holzer, Tingting Xiong, Ulrike Schnoor, Simon Melderis, Ina Kötter, Isabell Haase, Martin Krusche

Rheumatology International · 2026

DOI: 10.1007/s00296-026-06178-1 ↗

Kurz gefasst

Beobachtungsstudie in der Zeitschrift Rheumatology International, 2026. Verfasst von Phillip Kremer und 9 weiteren Autor:innen. Beteiligte Einrichtung: III. Department of Medicine, University Medical Center Hamburg-Eppendorf, Hamburg, Germany. Hauptthemen laut MEDLINE: Large Language Models, Connective Tissue Diseases, Benchmarking, Patient Education as Topic. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Universitätsklinikum Hamburg-Eppendorf (UKE).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

To co-develop disease-specific patient questions for connective tissue diseases (CTDs), compare patient/rheumatologist ratings of answers from language models (LLMs) versus Google Search, and quantify EULAR coverage of these patient-prioritized questions. In this prospective single-center observational study, reported in accordance with STROBE, patient advocacy groups curated 20 frequently asked questions (FAQs) for each CTD (systemic lupus erythematosus, idiopathic inflammatory myopathy, Sjögren disease, systemic sclerosis). Questions were submitted to Claude 4.0 Sonnet, ChatGPT-5, Gemini 2.5 Pro, and Google Search. Five patients per disease and five rheumatologists rated blinded outputs using forced-rank preference and 5-point Likert scales (patients: empathy, trustworthiness, comprehensibility; physicians: medical correctness, empathy, comprehensibility). Guideline mapping assessed whether EULAR recommendations addressed each question. All three LLMs answered 100% of questions (80/80), whereas Google Search answered 90% (72/80). Across CTDs, patients rated LLM outputs favorably for empathy (mean 1.40–2.77), trustworthiness (1.47–2.6), and comprehensibility (1.33–2.27), and rheumatologists for medical correctness (1.23–1.98). Patients and physicians most often ranked Gemini 2.5 Pro best overall (rank 1: 59% and 63%), and Google Search most often worst (rank 4: 55% and 57%). Cluster analyses showed no consistent differences across five clusters. Guideline mapping revealed substantial gaps: 40–55% of prioritized FAQs were not addressed by EULAR recommendations, with largest gaps in cluster E, life impact, psychosocial aspects and family planning (75–100% not addressed; 0% fully addressed). In this blinded evaluation, LLMs produced CTD FAQs responses patients perceived as empathic, trustworthy and rheumatologists rated medically correct, with Gemini 2.5 Pro most consistently preferred overall. However, the mismatch between prioritized questions and guideline coverage underscores the need for patient-centered, evidence-grounded information resources.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsConnective Tissue DiseasesBenchmarkingPatient Education as TopicHumansProspective StudiesPractice Guidelines as TopicFemaleEmpathyComprehensionRheumatologistsMiddle AgedMaleAdultAttitude of Health PersonnelHealth Knowledge, Attitudes, Practice

Stichwörter der Autor:innen

Artificial intelligenceConnective tissue diseasesConsumer health informationDigital healthLarge language modelsMyositisPatient carePatient questionsPatienteducation as topicSclerodermaSjogren's syndromeSjögren diseaseSystemic lupus erythematosusSystemic sclerosis

Autor:innen und Institutionen

  • III. Department of Medicine, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
  • Institute for Digital Medicine, Philipps-Universität Marburg, University Hospital of Giessen and Marburg, Baldingerstrasse, Marburg, Germany.
  • Rheumazentrum Lübeck, Lübeck, Germany.
  • Clinic for Rheumatology and Clinical Immunology, Auenlandklinik Bad Bramstedt, Bad Bramstedt, Germany.
  • III. Department of Medicine, University Medical Center Hamburg-Eppendorf, Hamburg, Germany. m.krusche@uke.de.

Förderung

  • Universitätsklinikum Hamburg-Eppendorf (UKE)

Zitieren

APA

Kremer, P., Knitza, J., Reichard, A., Holzer, M. T., Xiong, T., Schnoor, U., Melderis, S., Kötter, I., Haase, I., & Krusche, M. (2026). A patient-derived benchmark for evaluating large language models in connective tissue diseases: blinded multi-stakeholder assessment and guideline comparison. Rheumatology International. https://doi.org/10.1007/s00296-026-06178-1

Vancouver

Kremer P, Knitza J, Reichard A, Holzer MT, Xiong T, Schnoor U, et al. A patient-derived benchmark for evaluating large language models in connective tissue diseases: blinded multi-stakeholder assessment and guideline comparison. Rheumatology International. 2026. doi:10.1007/s00296-026-06178-1

BibTeX
@article{Kremer2026,
  author = {Kremer, Phillip and Knitza, Johannes and Reichard, Anna and Holzer, Marie Therese and Xiong, Tingting and Schnoor, Ulrike and Melderis, Simon and Kötter, Ina and Haase, Isabell and Krusche, Martin},
  title = {A patient-derived benchmark for evaluating large language models in connective tissue diseases: blinded multi-stakeholder assessment and guideline comparison},
  journal = {Rheumatology International},
  year = {2026},
  doi = {10.1007/s00296-026-06178-1}
}
Kostenlos starten →