EUVIMEDDescubrir EUVIMED
Open access

Evaluating the Efficacy of ChatGPT as a Patient Education Tool in Prostate Cancer: Multimetric Assessment

Damien Gibson iD, Stuart Jackson iD, Ramesh Shanmugasundaram iD, Ishith Seth iD, Adrian Siu iD, Nariman Ahmadi iD, Jonathan Kam iD, Nicholas Mehan iD, Ruban Thanigasalam iD, Nicola Jeffery iD, Manish I Patel iD, Scott Leslie iD

Journal of Medical Internet Research · 2024-08-14 · 26 · e55939

DOI: 10.2196/55939 ↗

At a glance

Journal article in Journal of Medical Internet Research, published online on 2024-08-14. Written by Damien Gibson and 11 co-authors. Institution: Department of Urology, Saint George Hospital, Kogarah, Australia. Major MEDLINE topics: Prostatic Neoplasms, Patient Education as Topic. Cited 126 times so far (Crossref); 56 references. The full text is freely available in PubMed Central.

ResumenSubjectsDatos bibliográficosCiteRelated studies

Abstract (original language)

Background

Artificial intelligence (AI) chatbots, such as ChatGPT, have made significant progress. These chatbots, particularly popular among health care professionals and patients, are transforming patient education and disease experience with personalized information. Accurate, timely patient education is crucial for informed decision-making, especially regarding prostate-specific antigen screening and treatment options. However, the accuracy and reliability of AI chatbots’ medical information must be rigorously evaluated. Studies testing ChatGPT’s knowledge of prostate cancer are emerging, but there is a need for ongoing evaluation to ensure the quality and safety of information provided to patients.

Objective

This study aims to evaluate the quality, accuracy, and readability of ChatGPT-4’s responses to common prostate cancer questions posed by patients.

Methods

Overall, 8 questions were formulated with an inductive approach based on information topics in peer-reviewed literature and Google Trends data. Adapted versions of the Patient Education Materials Assessment Tool for AI (PEMAT-AI), Global Quality Score, and DISCERN-AI tools were used by 4 independent reviewers to assess the quality of the AI responses. The 8 AI outputs were judged by 7 expert urologists, using an assessment framework developed to assess accuracy, safety, appropriateness, actionability, and effectiveness. The AI responses’ readability was assessed using established algorithms (Flesch Reading Ease score, Gunning Fog Index, Flesch-Kincaid Grade Level, The Coleman-Liau Index, and Simple Measure of Gobbledygook [SMOG] Index). A brief tool (Reference Assessment AI [REF-AI]) was developed to analyze the references provided by AI outputs, assessing for reference hallucination, relevance, and quality of references.

Results

The PEMAT-AI understandability score was very good (mean 79.44%, SD 10.44%), the DISCERN-AI rating was scored as “good” quality (mean 13.88, SD 0.93), and the Global Quality Score was high (mean 4.46/5, SD 0.50). Natural Language Assessment Tool for AI had pooled mean accuracy of 3.96 (SD 0.91), safety of 4.32 (SD 0.86), appropriateness of 4.45 (SD 0.81), actionability of 4.05 (SD 1.15), and effectiveness of 4.09 (SD 0.98). The readability algorithm consensus was “difficult to read” (Flesch Reading Ease score mean 45.97, SD 8.69; Gunning Fog Index mean 14.55, SD 4.79), averaging an 11th-grade reading level, equivalent to 15- to 17-year-olds (Flesch-Kincaid Grade Level mean 12.12, SD 4.34; The Coleman-Liau Index mean 12.75, SD 1.98; SMOG Index mean 11.06, SD 3.20). REF-AI identified 2 reference hallucinations, while the majority (28/30, 93%) of references appropriately supplemented the text. Most references (26/30, 86%) were from reputable government organizations, while a handful were direct citations from scientific literature.

Conclusions

Our analysis found that ChatGPT-4 provides generally good responses to common prostate cancer queries, making it a potentially valuable tool for patient education in prostate cancer care. Objective quality assessment tools indicated that the natural language processing outputs were generally reliable and appropriate, but there is room for improvement.

Subjects

MeSH terms (MEDLINE)

Prostatic NeoplasmsPatient Education as TopicHumansMaleArtificial Intelligence

Author keywords

AIAI chatbotsAI language modelChatGPTNLPantigen screeningartificial intelligencecancerdecision-makinghealth care professionalhealth care professionalslarge language modelmanmedical informationmen

Authors and institutions

  • Department of Urology, Saint George Hospital, Kogarah, Australia.
  • Faculty of Medicine, The University of New South Wales, Sydney, Australia.
  • Surgical Outcomes Research Centre, Sydney, Australia.
  • Faculty of Medicine, University of Sydney, Sydney, Australia.
  • Department of Surgery, Peninsula Health, Victoria, Australia.
  • Concord Institute of Academic Surgery, Concord Hospital, Sydney, Australia.

ORCID profiles: Damien Gibson, Stuart Jackson, Ramesh Shanmugasundaram, Ishith Seth, Adrian Siu, Nariman Ahmadi, Jonathan Kam, Nicholas Mehan, Ruban Thanigasalam, Nicola Jeffery, Manish I Patel, Scott Leslie

Cite

APA

Gibson, D., Jackson, S., Shanmugasundaram, R., Seth, I., Siu, A., Ahmadi, N., Kam, J., Mehan, N., Thanigasalam, R., Jeffery, N., Patel, M. I., & Leslie, S. (2024). Evaluating the Efficacy of ChatGPT as a Patient Education Tool in Prostate Cancer: Multimetric Assessment. Journal of Medical Internet Research, 26, e55939. https://doi.org/10.2196/55939

Vancouver

Gibson D, Jackson S, Shanmugasundaram R, Seth I, Siu A, Ahmadi N, et al. Evaluating the Efficacy of ChatGPT as a Patient Education Tool in Prostate Cancer: Multimetric Assessment. Journal of Medical Internet Research. 2024;26:e55939. doi:10.2196/55939

BibTeX
@article{Gibson2024,
  author = {Gibson, Damien and Jackson, Stuart and Shanmugasundaram, Ramesh and Seth, Ishith and Siu, Adrian and Ahmadi, Nariman and Kam, Jonathan and Mehan, Nicholas and Thanigasalam, Ruban and Jeffery, Nicola and Patel, Manish I and Leslie, Scott},
  title = {Evaluating the Efficacy of ChatGPT as a Patient Education Tool in Prostate Cancer: Multimetric Assessment},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e55939},
  doi = {10.2196/55939}
}
Start for free →