EUVIMEDEUVIMED entdecken
LeitlinieOpen Access · CC BY 4.0

Assessing ChatGPT-v4 for Guideline-Concordant Inflammatory Bowel Disease: Accuracy, Completeness, and Temporal Drift

Bewertung von ChatGPT-v4 auf Leitlinienkonformität bei entzündlicher Darmerkrankung: Genauigkeit, Vollständigkeit und zeitliche Drift

Oguz Ozturk iD, Mucahit Ergul iD, Yavuz Cagir iD, Ali Atay iD, Kadir Can Acun, Orhan Coskun, Ilyas Tenlik, Muhammed Bahaddin Durak, Ilhami Yuksel iD

Journal of Clinical Medicine · 29.06.2025 · 14 (13) · 4599

DOI: 10.3390/jcm14134599 ↗

Auf einen Blick

ChatGPT-4 wurde mit 102 ECCO-orientierten Fragen zu Colitis ulcerosa und Morbus Crohn geprüft. Die Genauigkeit war teils hoch, zeigte aber über 180 Tage Unterschiede.

Fragestellung

Wie genau und konsistent beantwortet ChatGPT-4 ECCO-konforme Fragen zu Diagnostik und Therapie von Colitis ulcerosa und Morbus Crohn?

Methode

102-Fragen-Querschnittsprüfung mit True/False- und Multiple-Choice-Fragen; Bewertung an Tag 1, Tag 15 und Tag 180 anhand von Likert-Skalen.

Ergebnisse

51 True/False-Items: 92,8 % zu Beginn, 92,8 % an Tag 15, 98,0 % an Tag 180. Multiple Choice: 90,2 % an Tag 1, 92,2 % an Tag 15, 84,3 % an Tag 180.

Einschränkungen

Die Zuverlässigkeit der Daten war suboptimal.

Schlussfolgerung der Autor:innen

ChatGPT-4 zeigt Potenzial als klinisches Entscheidungshilfesystem für UC und CD, weist aber zeitliche Variabilität und inkonsistente Aufgabenerfüllung auf.

Kernaussagen

  • True/False-Items: 92,8 % bis 98,0 % Genauigkeit
  • Multiple Choice: 90,2 % bis 84,3 % Genauigkeit
  • Zeitliche Variabilität und inkonsistente Aufgabenlösung

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

Kurz gefasst

Leitlinie in der Zeitschrift Journal of Clinical Medicine, online erschienen am 29.06.2025. Verfasst von Oguz Ozturk und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Gastroenterology, Ankara Bilkent City Hospital, Ankara 06170, Turkey. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 18 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

Auf einen BlickAbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background/Objectives: Chat Generative Pretrained Transformer (ChatGPT) is a useful resource for individuals working in the healthcare field. This paper will include descriptions of several ways in which ChatGPT-4 can achieve greater accuracy in its diagnosis and treatment plans for ulcerative colitis (UC) and Crohn’s disease (CD) by following the guidelines set out by the European Crohn’s and Colitis Organization (ECCO).

Methods

The survey, which comprised 102 questions, was developed to assess the precision and consistency of respondents’ responses regarding the UC and CD. The questionnaire incorporated true/false and multiple-choice questions, with the objective of simulating real-life scenarios and adhering to the ECCO guidelines. We employed Likert scales to assess the responses. The inquiries were put to ChatGPT-4 on the initial day, the 15th day, and the 180th day.

Results

The 51 true or false items demonstrated stability over a six-month period, with an initial accuracy of 92.8% at baseline, 92.8% on the 15th day, and peaked to 98.0% on the 180th day. This finding suggests a negligible effect size. The accuracy of the multiple-choice questions was initially 90.2% on Day 1, reached its highest point at 92.2% on Day 15, and then decreased to 84.3% on Day 180. However, the reliability of the data was found to be suboptimal, and the impact was deemed negligible. A modest, transient increase in performance was observed at 15 days, which subsequently diminished by 180 days, resulting in negligible effect sizes.

Conclusions

ChatGPT-4 demonstrates potential as a clinical decision support system for UC and CD, but its assessment is marked by temporal variability and the inconsistent execution of various tasks. Essential initiatives that should be carried out before involving artificial intelligence (AI) technology in IBD trials are routine revalidation, multi-rater comparisons, prompt standardization, and the cultivation of a comprehensive understanding of the model’s limitations.

Schlagwörter

Stichwörter der Autor:innen

ChatGPTartificial intelligenceclinical decision supportinflammatory bowel diseases

Autor:innen und Institutionen

  • Department of Gastroenterology, Ankara Bilkent City Hospital, Ankara 06170, Turkey.
  • Department of Gastroenterology, Ankara Yildirim Beyazit University Yenimahalle Training and Research Hospital, Ankara 06560, Turkey.
  • Department of Gastroenterology, Faculty of Medicine, Hacettepe University, Ankara 06100, Turkey.
  • Department of Gastroenterology, Faculty of Medicine, Ankara Yildirim Beyazit University, Ankara 06110, Turkey.

ORCID-Profile: Oguz Ozturk, Mucahit Ergul, Yavuz Cagir, Ali Atay, Ilhami Yuksel

Zitieren

APA

Ozturk, O., Ergul, M., Cagir, Y., Atay, A., Acun, K. C., Coskun, O., Tenlik, I., Durak, M. B., & Yuksel, I. (2025). Assessing ChatGPT-v4 for Guideline-Concordant Inflammatory Bowel Disease: Accuracy, Completeness, and Temporal Drift. Journal of Clinical Medicine, 14(13), 4599. https://doi.org/10.3390/jcm14134599

Vancouver

Ozturk O, Ergul M, Cagir Y, Atay A, Acun KC, Coskun O, et al. Assessing ChatGPT-v4 for Guideline-Concordant Inflammatory Bowel Disease: Accuracy, Completeness, and Temporal Drift. Journal of Clinical Medicine. 2025;14(13):4599. doi:10.3390/jcm14134599

BibTeX
@article{Ozturk2025,
  author = {Ozturk, Oguz and Ergul, Mucahit and Cagir, Yavuz and Atay, Ali and Acun, Kadir Can and Coskun, Orhan and Tenlik, Ilyas and Durak, Muhammed Bahaddin and Yuksel, Ilhami},
  title = {Assessing ChatGPT-v4 for Guideline-Concordant Inflammatory Bowel Disease: Accuracy, Completeness, and Temporal Drift},
  journal = {Journal of Clinical Medicine},
  year = {2025},
  volume = {14},
  number = {13},
  pages = {4599},
  doi = {10.3390/jcm14134599}
}
Kostenlos starten →