ChatGPT as a patient education tool in colorectal cancer—An in‐depth assessment of efficacy, quality and readability
ChatGPT als Instrument zur Patientenedukation bei kolorektalem Karzinom – eine detaillierte Bewertung von Wirksamkeit, Qualität und Lesbarkeit
Colorectal Disease · 2025-01 · 27 (1)
DOI: 10.1111/codi.17267 ↗Auf einen Blick
ChatGPT-4 wurde mit 8 Fragen zu kolorektalem Karzinom geprüft. Die Antworten waren überwiegend korrekt und sicher, aber schwer lesbar.
Fragestellung
Wie gut sind ChatGPT-4-Antworten zu kolorektalem Karzinom hinsichtlich Genauigkeit, Qualität und Lesbarkeit?
Methode
Bewertung von ChatGPT-4-Antworten auf 8 CRC-Fragen durch 8 kolorektale Chirurgen; Qualität mit PEMAT-AI, DISCERN-AI und GQS; Lesbarkeit mit FRE und GFI.
Ergebnisse
Die Antworten waren median korrekt 4,00, sicher 4,25, angemessen 4,00, umsetzbar 4,00 und effektiv 4,00. PEMAT-AI 71,43, DISCERN-AI 12,00, GQS 4,00; FRE 47,00, GFI 12,40.
Schlussfolgerung der Autor:innen
Die Autoren kommen zu dem Schluss, dass ChatGPT-4 sichere, aber unspezifische medizinische Informationen liefern kann und Potenzial als Patientenedukationshilfe hat. Verbesserte Lesbarkeit ist nötig.
Kernaussagen
- 8 CRC-Fragen wurden von 8 kolorektalen Chirurgen bewertet.
- Antworten: median korrekt 4,00 und sicher 4,25.
- Lesbarkeit erschwert: FRE 47,00, GFI 12,40.
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
At a glance
Evaluationsstudie in Colorectal Disease, published online on 2024-12-17. Written by Adrian H Y Siu and 10 co-authors. Institution: Concord Institute of Academic Surgery (CIAS), Concord Repatriation General Hospital, Concord, New South Wales, Australia. Major MEDLINE topics: Colorectal Neoplasms, Comprehension, Patient Education as Topic, Artificial Intelligence. Cited 29 times so far (Crossref); 41 references.
Topics: Selbstmanagement bei chronischen Erkrankungen, Künstliche Intelligenz in der Pflege
Abstract (original language)
Aim
Artificial intelligence (AI) chatbots such as Chat Generative Pretrained Transformer‐4 (ChatGPT‐4) have made significant strides in generating human‐like responses. Trained on an extensive corpus of medical literature, ChatGPT‐4 has the potential to augment patient education materials. These chatbots may be beneficial to populations considering a diagnosis of colorectal cancer (CRC). However, the accuracy and quality of patient education materials are crucial for informed decision‐making. Given workforce demands impacting holistic care, AI chatbots can bridge gaps in CRC information, reaching wider demographics and crossing language barriers. However, rigorous evaluation is essential to ensure accuracy, quality and readability. Therefore, this study aims to evaluate the efficacy, quality and readability of answers generated by ChatGPT‐4 on CRC, utilizing patient‐style question prompts.
Method
To evaluate ChatGPT‐4, eight CRC‐related questions were derived using peer‐reviewed literature and Google Trends. Eight colorectal surgeons evaluated AI responses for accuracy, safety, appropriateness, actionability and effectiveness. Quality was assessed using validated tools: the Patient Education Materials Assessment Tool (PEMAT‐AI), modified DISCERN (DISCERN‐AI) and Global Quality Score (GQS). A number of readability assessments were measured including Flesch Reading Ease (FRE) and the Gunning Fog Index (GFI).
Results
The responses were generally accurate (median 4.00), safe (4.25), appropriate (4.00), actionable (4.00) and effective (4.00). Quality assessments rated PEMAT‐AI as ‘very good’ (71.43), DISCERN‐AI as ‘fair’ (12.00) and GQS as ‘high’ (4.00). Readability scores indicated difficulty (FRE 47.00, GFI 12.40), suggesting a higher educational level was required.
Conclusion
This study concludes that ChatGPT‐4 is capable of providing safe but nonspecific medical information, suggesting its potential as a patient education aid. However, enhancements in readability through contextual prompting and fine‐tuning techniques are required before considering implementation into clinical practice.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Concord Institute of Academic Surgery (CIAS), Concord Repatriation General Hospital, Concord, New South Wales, Australia.
- Faculty of Medicine and Health, Central Clinical School, The University of Sydney, Sydney, New South Wales, Australia.
- Surgical Outcomes Research Centre (SOuRCe), Royal Prince Alfred Hospital, Camperdown, New South Wales, Australia.
- Division of Colorectal Surgery, Department of Surgery, Concord Repatriation General Hospital, Sydney, New South Wales, Australia.
- Department of Colorectal Surgery, Bankstown-Lidcombe Hospital, South Western Sydney Local Health District, Sydney, New South Wales, Australia.
- School of Clinical Medicine, University of new South Wales, Sydney, New South Wales, Australia.
ORCID profiles: Adrian H Y Siu, Allan Kwok, Mifanwy Reece, Matthew J F X Rickard
Cite
APA
Siu, A. H. Y., Gibson, D. P., Chiu, C., Kwok, A., Irwin, M., Christie, A., Koh, C. E., Keshava, A., Reece, M., Suen, M., & Rickard, M. J. F. X. (2025). ChatGPT as a patient education tool in colorectal cancer—An in‐depth assessment of efficacy, quality and readability. Colorectal Disease, 27(1). https://doi.org/10.1111/codi.17267
Vancouver
Siu AHY, Gibson DP, Chiu C, Kwok A, Irwin M, Christie A, et al. ChatGPT as a patient education tool in colorectal cancer—An in‐depth assessment of efficacy, quality and readability. Colorectal Disease. 2025;27(1). doi:10.1111/codi.17267
BibTeX
@article{Siu2025,
author = {Siu, Adrian H Y and Gibson, Damien P and Chiu, Chris and Kwok, Allan and Irwin, Matt and Christie, Adam and Koh, Cherry E and Keshava, Anil and Reece, Mifanwy and Suen, Michael and Rickard, Matthew J F X},
title = {ChatGPT as a patient education tool in colorectal cancer—An in‐depth assessment of efficacy, quality and readability},
journal = {Colorectal Disease},
year = {2025},
volume = {27},
number = {1},
doi = {10.1111/codi.17267}
}
EUVIMED