Assessing ChatGPT-v4 for Guideline-Concordant Inflammatory Bowel Disease: Accuracy, Completeness, and Temporal Drift
Bewertung von ChatGPT-v4 auf Leitlinienkonformität bei entzündlicher Darmerkrankung: Genauigkeit, Vollständigkeit und zeitliche Drift
Journal of Clinical Medicine · 29.06.2025 · 14 (13) · 4599
DOI: 10.3390/jcm14134599 ↗Auf einen Blick
ChatGPT-4 wurde mit 102 ECCO-orientierten Fragen zu Colitis ulcerosa und Morbus Crohn geprüft. Die Genauigkeit war teils hoch, zeigte aber über 180 Tage Unterschiede.
Fragestellung
Wie genau und konsistent beantwortet ChatGPT-4 ECCO-konforme Fragen zu Diagnostik und Therapie von Colitis ulcerosa und Morbus Crohn?
Methode
102-Fragen-Querschnittsprüfung mit True/False- und Multiple-Choice-Fragen; Bewertung an Tag 1, Tag 15 und Tag 180 anhand von Likert-Skalen.
Ergebnisse
51 True/False-Items: 92,8 % zu Beginn, 92,8 % an Tag 15, 98,0 % an Tag 180. Multiple Choice: 90,2 % an Tag 1, 92,2 % an Tag 15, 84,3 % an Tag 180.
Einschränkungen
Die Zuverlässigkeit der Daten war suboptimal.
Schlussfolgerung der Autor:innen
ChatGPT-4 zeigt Potenzial als klinisches Entscheidungshilfesystem für UC und CD, weist aber zeitliche Variabilität und inkonsistente Aufgabenerfüllung auf.
Kernaussagen
- True/False-Items: 92,8 % bis 98,0 % Genauigkeit
- Multiple Choice: 90,2 % bis 84,3 % Genauigkeit
- Zeitliche Variabilität und inkonsistente Aufgabenlösung
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Leitlinie in der Zeitschrift Journal of Clinical Medicine, online erschienen am 29.06.2025. Verfasst von Oguz Ozturk und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Gastroenterology, Ankara Bilkent City Hospital, Ankara 06170, Turkey. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 18 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background/Objectives: Chat Generative Pretrained Transformer (ChatGPT) is a useful resource for individuals working in the healthcare field. This paper will include descriptions of several ways in which ChatGPT-4 can achieve greater accuracy in its diagnosis and treatment plans for ulcerative colitis (UC) and Crohn’s disease (CD) by following the guidelines set out by the European Crohn’s and Colitis Organization (ECCO).
Methods
The survey, which comprised 102 questions, was developed to assess the precision and consistency of respondents’ responses regarding the UC and CD. The questionnaire incorporated true/false and multiple-choice questions, with the objective of simulating real-life scenarios and adhering to the ECCO guidelines. We employed Likert scales to assess the responses. The inquiries were put to ChatGPT-4 on the initial day, the 15th day, and the 180th day.
Results
The 51 true or false items demonstrated stability over a six-month period, with an initial accuracy of 92.8% at baseline, 92.8% on the 15th day, and peaked to 98.0% on the 180th day. This finding suggests a negligible effect size. The accuracy of the multiple-choice questions was initially 90.2% on Day 1, reached its highest point at 92.2% on Day 15, and then decreased to 84.3% on Day 180. However, the reliability of the data was found to be suboptimal, and the impact was deemed negligible. A modest, transient increase in performance was observed at 15 days, which subsequently diminished by 180 days, resulting in negligible effect sizes.
Conclusions
ChatGPT-4 demonstrates potential as a clinical decision support system for UC and CD, but its assessment is marked by temporal variability and the inconsistent execution of various tasks. Essential initiatives that should be carried out before involving artificial intelligence (AI) technology in IBD trials are routine revalidation, multi-rater comparisons, prompt standardization, and the cultivation of a comprehensive understanding of the model’s limitations.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Gastroenterology, Ankara Bilkent City Hospital, Ankara 06170, Turkey.
- Department of Gastroenterology, Ankara Yildirim Beyazit University Yenimahalle Training and Research Hospital, Ankara 06560, Turkey.
- Department of Gastroenterology, Faculty of Medicine, Hacettepe University, Ankara 06100, Turkey.
- Department of Gastroenterology, Faculty of Medicine, Ankara Yildirim Beyazit University, Ankara 06110, Turkey.
ORCID-Profile: Oguz Ozturk, Mucahit Ergul, Yavuz Cagir, Ali Atay, Ilhami Yuksel
Zitieren
APA
Ozturk, O., Ergul, M., Cagir, Y., Atay, A., Acun, K. C., Coskun, O., Tenlik, I., Durak, M. B., & Yuksel, I. (2025). Assessing ChatGPT-v4 for Guideline-Concordant Inflammatory Bowel Disease: Accuracy, Completeness, and Temporal Drift. Journal of Clinical Medicine, 14(13), 4599. https://doi.org/10.3390/jcm14134599
Vancouver
Ozturk O, Ergul M, Cagir Y, Atay A, Acun KC, Coskun O, et al. Assessing ChatGPT-v4 for Guideline-Concordant Inflammatory Bowel Disease: Accuracy, Completeness, and Temporal Drift. Journal of Clinical Medicine. 2025;14(13):4599. doi:10.3390/jcm14134599
BibTeX
@article{Ozturk2025,
author = {Ozturk, Oguz and Ergul, Mucahit and Cagir, Yavuz and Atay, Ali and Acun, Kadir Can and Coskun, Orhan and Tenlik, Ilyas and Durak, Muhammed Bahaddin and Yuksel, Ilhami},
title = {Assessing ChatGPT-v4 for Guideline-Concordant Inflammatory Bowel Disease: Accuracy, Completeness, and Temporal Drift},
journal = {Journal of Clinical Medicine},
year = {2025},
volume = {14},
number = {13},
pages = {4599},
doi = {10.3390/jcm14134599}
}
EUVIMED