EUVIMEDEUVIMED entdecken
Open Access

Evaluating the Efficacy of ChatGPT as a Patient Education Tool in Prostate Cancer: Multimetric Assessment

Damien Gibson iD, Stuart Jackson iD, Ramesh Shanmugasundaram iD, Ishith Seth iD, Adrian Siu iD, Nariman Ahmadi iD, Jonathan Kam iD, Nicholas Mehan iD, Ruban Thanigasalam iD, Nicola Jeffery iD, Manish I Patel iD, Scott Leslie iD

Journal of Medical Internet Research · 14.08.2024 · 26 · e55939

DOI: 10.2196/55939 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 14.08.2024. Verfasst von Damien Gibson und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Urology, Saint George Hospital, Kogarah, Australia. Hauptthemen laut MEDLINE: Prostatic Neoplasms, Patient Education as Topic. Die Arbeit wurde bisher 126-mal zitiert (Crossref) und nennt 56 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Artificial intelligence (AI) chatbots, such as ChatGPT, have made significant progress. These chatbots, particularly popular among health care professionals and patients, are transforming patient education and disease experience with personalized information. Accurate, timely patient education is crucial for informed decision-making, especially regarding prostate-specific antigen screening and treatment options. However, the accuracy and reliability of AI chatbots’ medical information must be rigorously evaluated. Studies testing ChatGPT’s knowledge of prostate cancer are emerging, but there is a need for ongoing evaluation to ensure the quality and safety of information provided to patients.

Objective

This study aims to evaluate the quality, accuracy, and readability of ChatGPT-4’s responses to common prostate cancer questions posed by patients.

Methods

Overall, 8 questions were formulated with an inductive approach based on information topics in peer-reviewed literature and Google Trends data. Adapted versions of the Patient Education Materials Assessment Tool for AI (PEMAT-AI), Global Quality Score, and DISCERN-AI tools were used by 4 independent reviewers to assess the quality of the AI responses. The 8 AI outputs were judged by 7 expert urologists, using an assessment framework developed to assess accuracy, safety, appropriateness, actionability, and effectiveness. The AI responses’ readability was assessed using established algorithms (Flesch Reading Ease score, Gunning Fog Index, Flesch-Kincaid Grade Level, The Coleman-Liau Index, and Simple Measure of Gobbledygook [SMOG] Index). A brief tool (Reference Assessment AI [REF-AI]) was developed to analyze the references provided by AI outputs, assessing for reference hallucination, relevance, and quality of references.

Results

The PEMAT-AI understandability score was very good (mean 79.44%, SD 10.44%), the DISCERN-AI rating was scored as “good” quality (mean 13.88, SD 0.93), and the Global Quality Score was high (mean 4.46/5, SD 0.50). Natural Language Assessment Tool for AI had pooled mean accuracy of 3.96 (SD 0.91), safety of 4.32 (SD 0.86), appropriateness of 4.45 (SD 0.81), actionability of 4.05 (SD 1.15), and effectiveness of 4.09 (SD 0.98). The readability algorithm consensus was “difficult to read” (Flesch Reading Ease score mean 45.97, SD 8.69; Gunning Fog Index mean 14.55, SD 4.79), averaging an 11th-grade reading level, equivalent to 15- to 17-year-olds (Flesch-Kincaid Grade Level mean 12.12, SD 4.34; The Coleman-Liau Index mean 12.75, SD 1.98; SMOG Index mean 11.06, SD 3.20). REF-AI identified 2 reference hallucinations, while the majority (28/30, 93%) of references appropriately supplemented the text. Most references (26/30, 86%) were from reputable government organizations, while a handful were direct citations from scientific literature.

Conclusions

Our analysis found that ChatGPT-4 provides generally good responses to common prostate cancer queries, making it a potentially valuable tool for patient education in prostate cancer care. Objective quality assessment tools indicated that the natural language processing outputs were generally reliable and appropriate, but there is room for improvement.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Prostatic NeoplasmsPatient Education as TopicHumansMaleArtificial Intelligence

Stichwörter der Autor:innen

AIAI chatbotsAI language modelChatGPTNLPantigen screeningartificial intelligencecancerdecision-makinghealth care professionalhealth care professionalslarge language modelmanmedical informationmen

Autor:innen und Institutionen

  • Department of Urology, Saint George Hospital, Kogarah, Australia.
  • Faculty of Medicine, The University of New South Wales, Sydney, Australia.
  • Surgical Outcomes Research Centre, Sydney, Australia.
  • Faculty of Medicine, University of Sydney, Sydney, Australia.
  • Department of Surgery, Peninsula Health, Victoria, Australia.
  • Concord Institute of Academic Surgery, Concord Hospital, Sydney, Australia.

ORCID-Profile: Damien Gibson, Stuart Jackson, Ramesh Shanmugasundaram, Ishith Seth, Adrian Siu, Nariman Ahmadi, Jonathan Kam, Nicholas Mehan, Ruban Thanigasalam, Nicola Jeffery, Manish I Patel, Scott Leslie

Zitieren

APA

Gibson, D., Jackson, S., Shanmugasundaram, R., Seth, I., Siu, A., Ahmadi, N., Kam, J., Mehan, N., Thanigasalam, R., Jeffery, N., Patel, M. I., & Leslie, S. (2024). Evaluating the Efficacy of ChatGPT as a Patient Education Tool in Prostate Cancer: Multimetric Assessment. Journal of Medical Internet Research, 26, e55939. https://doi.org/10.2196/55939

Vancouver

Gibson D, Jackson S, Shanmugasundaram R, Seth I, Siu A, Ahmadi N, et al. Evaluating the Efficacy of ChatGPT as a Patient Education Tool in Prostate Cancer: Multimetric Assessment. Journal of Medical Internet Research. 2024;26:e55939. doi:10.2196/55939

BibTeX
@article{Gibson2024,
  author = {Gibson, Damien and Jackson, Stuart and Shanmugasundaram, Ramesh and Seth, Ishith and Siu, Adrian and Ahmadi, Nariman and Kam, Jonathan and Mehan, Nicholas and Thanigasalam, Ruban and Jeffery, Nicola and Patel, Manish I and Leslie, Scott},
  title = {Evaluating the Efficacy of ChatGPT as a Patient Education Tool in Prostate Cancer: Multimetric Assessment},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e55939},
  doi = {10.2196/55939}
}
Kostenlos starten →