Development and Nationwide Multicentre Evaluation of Guideline-Grounded Large Language Model Chatbots to Support Patient Self-Management and Education in Rheumatology
Journal of Medical Systems · 05.10.2026 · 50 (1)
DOI: 10.1007/s10916-026-02470-6 ↗Kurz gefasst
Multizentrische Studie in der Zeitschrift Journal of Medical Systems, erschienen am 05.10.2026. Verfasst von Tim Wilhelmi und 21 weiteren Autor:innen. Beteiligte Einrichtung: Department of Rheumatology, Clinical Immunology, Osteology and Physical Medicine, Justus Liebig University Giessen, Campus Kerckhoff, Bad Nauheim, Ge…. Hauptthemen laut MEDLINE: Patient Education as Topic, Self-Management, Rheumatology, Rheumatic Diseases. Die Arbeit nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Horizon Europe grant for the SPIDERR project, Deutsche Rheumastiftung.
Themen: Künstliche Intelligenz in der Pflege, Selbstmanagement bei chronischen Erkrankungen
Abstract (Originalsprache)
Patients with rheumatic diseases have persistent information needs that are not fully addressed in routine care. We developed and evaluated guideline-grounded, large language model (LLM) chatbots to support patient self-management and education in rheumatology. Ten disease-specific chatbots based on German guidelines were co-developed and deployed through 13 rheumatology centres and six patient organisations. Chatbot users rated responses and completed a questionnaire. User questions, feedback, and response characteristics were analysed using category-based coding and a six-dimensional LLM-as-a-judge assessment, with LLM-based ratings compared with rheumatologist ratings in random subsets. Between September 2025 and January 2026, 6291 questions were recorded. Thirteen question categories were identified, most commonly disease-specific questions (50.2%), medication and monitoring (37.3%) and diagnostics (28.2%). The chatbots were unable to answer in 263 interactions (4.2%). Of 2671 responses rated by users, 2481 (92.9%) received a positive rating. Insufficient detail was the most common reason for negative ratings (125/190, 65.8%). Among 602 questionnaire respondents, 84.6% reported that the chatbot was easy to use, 84.1% that answers were easy to understand, and 80.2% that it was a useful addition to patient education. In the LLM-based evaluation, 95.3% of answers were rated as completely safe and 79.1% as completely correct. Guideline adherence was assessed separately, with 45.0% rated as fully adherent; agreement with physician assessment was weak. Guideline-grounded chatbots received predominantly positive user feedback in real-world use, while LLM-based evaluation suggested that most responses were safe and correct. User questions and feedback may help guide iterative improvements to source content and patient education materials. Further studies are needed to evaluate educational effectiveness and independently validate response quality and clinical safety.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Rheumatology, Clinical Immunology, Osteology and Physical Medicine, Justus Liebig University Giessen, Campus Kerckhoff, Bad Nauheim, Germany.
- Division of Rheumatology, Paracelsus Medical University, Klinikum Nürnberg, Nürnberg, Germany.
- School of Medicine, Institute for Digital Medicine, Philipps-Universität Marburg, Marburg, Germany.
- Zentrum für Telemedizin, Bad Kissingen, Germany.
- Rheumapraxis an der Hase, Osnabrück, Germany.
- Division of Rheumatology, III. Department of Medicine, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
Förderung
- Horizon Europe grant for the SPIDERR project
- Deutsche Rheumastiftung
Zitieren
APA
Wilhelmi, T., Bartsch, V., Platt, M., Rashid, A., Hornig, J., Krusche, M., Hueber, A. J., Fink, D., Pfeil, A., Dischereit, G., Holzer, M. T., Drott, U., Aries, P., Müller, M., Böhm, P., Morf, H., Labinsky, H., Mühlensiepen, F., Benavent, D., …, & Knitza, J. (2026). Development and Nationwide Multicentre Evaluation of Guideline-Grounded Large Language Model Chatbots to Support Patient Self-Management and Education in Rheumatology. Journal of Medical Systems. https://doi.org/10.1007/s10916-026-02470-6
Vancouver
Wilhelmi T, Bartsch V, Platt M, Rashid A, Hornig J, Krusche M, et al. Development and Nationwide Multicentre Evaluation of Guideline-Grounded Large Language Model Chatbots to Support Patient Self-Management and Education in Rheumatology. Journal of Medical Systems. 2026. doi:10.1007/s10916-026-02470-6
BibTeX
@article{Wilhelmi2026,
author = {Wilhelmi, Tim and Bartsch, Vanessa and Platt, Marius and Rashid, Asarnusch and Hornig, Johannes and Krusche, Martin and Hueber, Axel J and Fink, Daniel and Pfeil, Alexander and Dischereit, Gabriel and Holzer, Marie-Therese and Drott, Ulrich and Aries, Peer and Müller, Max and Böhm, Peter and Morf, Harriet and Labinsky, Hannah and Mühlensiepen, Felix and Benavent, Diego and Kuhn, Sebastian and Klemm, Philipp and Knitza, Johannes},
title = {Development and Nationwide Multicentre Evaluation of Guideline-Grounded Large Language Model Chatbots to Support Patient Self-Management and Education in Rheumatology},
journal = {Journal of Medical Systems},
year = {2026},
doi = {10.1007/s10916-026-02470-6}
}
EUVIMED