EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Dr. Google to Dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitis

Yazid K Ghanem iD, Armaun D Rouhi, Ammr Al-Houssan, Zena Saleh, Matthew C Moccia, Hansa Joshi, Kristoffel R Dumon, Young Hong, Francis Spitz, Amit R Joshi, Michael Kwiatt

Surgical Endoscopy · 2024-05 · 38 (5) · 2887-2893

DOI: 10.1007/s00464-024-10739-5 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Surgical Endoscopy, online erschienen am 05.03.2024. Verfasst von Yazid K Ghanem und 10 weiteren Autor:innen. Beteiligte Einrichtung: Department of Surgery, Cooper University Hospital, 3 Cooper Plaza, Suite 411, Camden, NJ, 08103, USA. ghanem-yazidkhalilmo@cooperhealth.edu. Hauptthemen laut MEDLINE: Appendicitis, Artificial Intelligence. Die Arbeit wurde bisher 102-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

Generative artificial intelligence (AI) chatbots have recently been posited as potential sources of online medical information for patients making medical decisions. Existing online patient-oriented medical information has repeatedly been shown to be of variable quality and difficult readability. Therefore, we sought to evaluate the content and quality of AI-generated medical information on acute appendicitis.

Methods

A modified DISCERN assessment tool, comprising 16 distinct criteria each scored on a 5-point Likert scale (score range 16–80), was used to assess AI-generated content. Readability was determined using the Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) scores. Four popular chatbots, ChatGPT-3.5 and ChatGPT-4, Bard, and Claude-2, were prompted to generate medical information about appendicitis. Three investigators independently scored the generated texts blinded to the identity of the AI platforms.

Results

ChatGPT-3.5, ChatGPT-4, Bard, and Claude-2 had overall mean (SD) quality scores of 60.7 (1.2), 62.0 (1.0), 62.3 (1.2), and 51.3 (2.3), respectively, on a scale of 16–80. Inter-rater reliability was 0.81, 0.75, 0.81, and 0.72, respectively, indicating substantial agreement. Claude-2 demonstrated a significantly lower mean quality score compared to ChatGPT-4 ( p = 0.001), ChatGPT-3.5 ( p = 0.005), and Bard ( p = 0.001). Bard was the only AI platform that listed verifiable sources, while Claude-2 provided fabricated sources. All chatbots except for Claude-2 advised readers to consult a physician if experiencing symptoms. Regarding readability, FKGL and FRE scores of ChatGPT-3.5, ChatGPT-4, Bard, and Claude-2 were 14.6 and 23.8, 11.9 and 33.9, 8.6 and 52.8, 11.0 and 36.6, respectively, indicating difficulty readability at a college reading skill level.

Conclusion

AI-generated medical information on appendicitis scored favorably upon quality assessment, but most either fabricated sources or did not provide any altogether. Additionally, overall readability far exceeded recommended levels for the public. Generative AI platforms demonstrate measured potential for patient education and engagement about appendicitis.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

AppendicitisArtificial IntelligenceHumansComprehensionInternetConsumer Health InformationPatient Education as Topic

Stichwörter der Autor:innen

AppendicitisArtificial intelligenceChatGPTHealth literacyLarge language modelsOnline medical information

Autor:innen und Institutionen

  • Department of Surgery, Cooper University Hospital, 3 Cooper Plaza, Suite 411, Camden, NJ, 08103, USA. ghanem-yazidkhalilmo@cooperhealth.edu.
  • Cooper Medical School of Rowan University, Camden, NJ, USA. ghanem-yazidkhalilmo@cooperhealth.edu.
  • Department of Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.
  • Department of Surgery, University of Connecticut, Hartford, CT, USA.
  • Department of Surgery, Cooper University Hospital, 3 Cooper Plaza, Suite 411, Camden, NJ, 08103, USA.
  • Cooper Medical School of Rowan University, Camden, NJ, USA.

ORCID-Profile: Yazid K Ghanem

Zitieren

APA

Ghanem, Y. K., Rouhi, A. D., Al-Houssan, A., Saleh, Z., Moccia, M. C., Joshi, H., Dumon, K. R., Hong, Y., Spitz, F., Joshi, A. R., & Kwiatt, M. (2024). Dr. Google to Dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitis. Surgical Endoscopy, 38(5), 2887-2893. https://doi.org/10.1007/s00464-024-10739-5

Vancouver

Ghanem YK, Rouhi AD, Al-Houssan A, Saleh Z, Moccia MC, Joshi H, et al. Dr. Google to Dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitis. Surgical Endoscopy. 2024;38(5):2887-2893. doi:10.1007/s00464-024-10739-5

BibTeX
@article{Ghanem2024,
  author = {Ghanem, Yazid K and Rouhi, Armaun D and Al-Houssan, Ammr and Saleh, Zena and Moccia, Matthew C and Joshi, Hansa and Dumon, Kristoffel R and Hong, Young and Spitz, Francis and Joshi, Amit R and Kwiatt, Michael},
  title = {Dr. Google to Dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitis},
  journal = {Surgical Endoscopy},
  year = {2024},
  volume = {38},
  number = {5},
  pages = {2887--2893},
  doi = {10.1007/s00464-024-10739-5}
}
Kostenlos starten →