EUVIMEDAtklāt EUVIMED
Evaluationsstudie

ChatGPT as a patient education tool in colorectal cancer—An in‐depth assessment of efficacy, quality and readability

ChatGPT als Instrument zur Patientenedukation bei kolorektalem Karzinom – eine detaillierte Bewertung von Wirksamkeit, Qualität und Lesbarkeit

Adrian H Y Siu iD, Damien P Gibson, Chris Chiu, Allan Kwok iD, Matt Irwin, Adam Christie, Cherry E Koh, Anil Keshava, Mifanwy Reece iD, Michael Suen, Matthew J F X Rickard iD

Colorectal Disease · 2025-01 · 27 (1)

DOI: 10.1111/codi.17267 ↗

Auf einen Blick

ChatGPT-4 wurde mit 8 Fragen zu kolorektalem Karzinom geprüft. Die Antworten waren überwiegend korrekt und sicher, aber schwer lesbar.

Fragestellung

Wie gut sind ChatGPT-4-Antworten zu kolorektalem Karzinom hinsichtlich Genauigkeit, Qualität und Lesbarkeit?

Methode

Bewertung von ChatGPT-4-Antworten auf 8 CRC-Fragen durch 8 kolorektale Chirurgen; Qualität mit PEMAT-AI, DISCERN-AI und GQS; Lesbarkeit mit FRE und GFI.

Ergebnisse

Die Antworten waren median korrekt 4,00, sicher 4,25, angemessen 4,00, umsetzbar 4,00 und effektiv 4,00. PEMAT-AI 71,43, DISCERN-AI 12,00, GQS 4,00; FRE 47,00, GFI 12,40.

Schlussfolgerung der Autor:innen

Die Autoren kommen zu dem Schluss, dass ChatGPT-4 sichere, aber unspezifische medizinische Informationen liefern kann und Potenzial als Patientenedukationshilfe hat. Verbesserte Lesbarkeit ist nötig.

Kernaussagen

  • 8 CRC-Fragen wurden von 8 kolorektalen Chirurgen bewertet.
  • Antworten: median korrekt 4,00 und sicher 4,25.
  • Lesbarkeit erschwert: FRE 47,00, GFI 12,40.

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Evaluationsstudie in Colorectal Disease, published online on 2024-12-17. Written by Adrian H Y Siu and 10 co-authors. Institution: Concord Institute of Academic Surgery (CIAS), Concord Repatriation General Hospital, Concord, New South Wales, Australia. Major MEDLINE topics: Colorectal Neoplasms, Comprehension, Patient Education as Topic, Artificial Intelligence. Cited 29 times so far (Crossref); 41 references.

Auf einen BlickAbstraktsSubjectsBibliogrāfiskie datiCiteRelated studies

Abstract (original language)

Aim

Artificial intelligence (AI) chatbots such as Chat Generative Pretrained Transformer‐4 (ChatGPT‐4) have made significant strides in generating human‐like responses. Trained on an extensive corpus of medical literature, ChatGPT‐4 has the potential to augment patient education materials. These chatbots may be beneficial to populations considering a diagnosis of colorectal cancer (CRC). However, the accuracy and quality of patient education materials are crucial for informed decision‐making. Given workforce demands impacting holistic care, AI chatbots can bridge gaps in CRC information, reaching wider demographics and crossing language barriers. However, rigorous evaluation is essential to ensure accuracy, quality and readability. Therefore, this study aims to evaluate the efficacy, quality and readability of answers generated by ChatGPT‐4 on CRC, utilizing patient‐style question prompts.

Method

To evaluate ChatGPT‐4, eight CRC‐related questions were derived using peer‐reviewed literature and Google Trends. Eight colorectal surgeons evaluated AI responses for accuracy, safety, appropriateness, actionability and effectiveness. Quality was assessed using validated tools: the Patient Education Materials Assessment Tool (PEMAT‐AI), modified DISCERN (DISCERN‐AI) and Global Quality Score (GQS). A number of readability assessments were measured including Flesch Reading Ease (FRE) and the Gunning Fog Index (GFI).

Results

The responses were generally accurate (median 4.00), safe (4.25), appropriate (4.00), actionable (4.00) and effective (4.00). Quality assessments rated PEMAT‐AI as ‘very good’ (71.43), DISCERN‐AI as ‘fair’ (12.00) and GQS as ‘high’ (4.00). Readability scores indicated difficulty (FRE 47.00, GFI 12.40), suggesting a higher educational level was required.

Conclusion

This study concludes that ChatGPT‐4 is capable of providing safe but nonspecific medical information, suggesting its potential as a patient education aid. However, enhancements in readability through contextual prompting and fine‐tuning techniques are required before considering implementation into clinical practice.

Subjects

MeSH terms (MEDLINE)

Colorectal NeoplasmsComprehensionPatient Education as TopicArtificial IntelligenceHumansHealth Literacy

Author keywords

artificial intelligencecolorectal cancereducation modelspatient education

Authors and institutions

  • Concord Institute of Academic Surgery (CIAS), Concord Repatriation General Hospital, Concord, New South Wales, Australia.
  • Faculty of Medicine and Health, Central Clinical School, The University of Sydney, Sydney, New South Wales, Australia.
  • Surgical Outcomes Research Centre (SOuRCe), Royal Prince Alfred Hospital, Camperdown, New South Wales, Australia.
  • Division of Colorectal Surgery, Department of Surgery, Concord Repatriation General Hospital, Sydney, New South Wales, Australia.
  • Department of Colorectal Surgery, Bankstown-Lidcombe Hospital, South Western Sydney Local Health District, Sydney, New South Wales, Australia.
  • School of Clinical Medicine, University of new South Wales, Sydney, New South Wales, Australia.

ORCID profiles: Adrian H Y Siu, Allan Kwok, Mifanwy Reece, Matthew J F X Rickard

Cite

APA

Siu, A. H. Y., Gibson, D. P., Chiu, C., Kwok, A., Irwin, M., Christie, A., Koh, C. E., Keshava, A., Reece, M., Suen, M., & Rickard, M. J. F. X. (2025). ChatGPT as a patient education tool in colorectal cancer—An in‐depth assessment of efficacy, quality and readability. Colorectal Disease, 27(1). https://doi.org/10.1111/codi.17267

Vancouver

Siu AHY, Gibson DP, Chiu C, Kwok A, Irwin M, Christie A, et al. ChatGPT as a patient education tool in colorectal cancer—An in‐depth assessment of efficacy, quality and readability. Colorectal Disease. 2025;27(1). doi:10.1111/codi.17267

BibTeX
@article{Siu2025,
  author = {Siu, Adrian H Y and Gibson, Damien P and Chiu, Chris and Kwok, Allan and Irwin, Matt and Christie, Adam and Koh, Cherry E and Keshava, Anil and Reece, Mifanwy and Suen, Michael and Rickard, Matthew J F X},
  title = {ChatGPT as a patient education tool in colorectal cancer—An in‐depth assessment of efficacy, quality and readability},
  journal = {Colorectal Disease},
  year = {2025},
  volume = {27},
  number = {1},
  doi = {10.1111/codi.17267}
}
Start for free →