EUVIMEDEUVIMED entdecken
LeitlinieOpen Access

Reliability of Medical Information Provided by ChatGPT: Assessment Against Clinical Guidelines and Patient Information Quality Instrument

Harriet Louise Walker iD, Shahi Ghani iD, Christoph Kuemmerli iD, Christian Andreas Nebiker iD, Beat Peter Müller iD, Dimitri Aristotle Raptis iD, Sebastian Manuel Staubli iD

Journal of Medical Internet Research · 30.06.2023 · 25 · e47479

DOI: 10.2196/47479 ↗

Kurz gefasst

Leitlinie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 30.06.2023. Verfasst von Harriet Louise Walker und 6 weiteren Autor:innen. Beteiligte Einrichtung: Royal Free London NHS Foundation Trust, London, United Kingdom. Hauptthemen laut MEDLINE: Artificial Intelligence, Health Personnel. Die Arbeit wurde bisher 375-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

ChatGPT-4 is the latest release of a novel artificial intelligence (AI) chatbot able to answer freely formulated and complex questions. In the near future, ChatGPT could become the new standard for health care professionals and patients to access medical information. However, little is known about the quality of medical information provided by the AI.

Objective

We aimed to assess the reliability of medical information provided by ChatGPT.

Methods

Medical information provided by ChatGPT-4 on the 5 hepato-pancreatico-biliary (HPB) conditions with the highest global disease burden was measured with the Ensuring Quality Information for Patients (EQIP) tool. The EQIP tool is used to measure the quality of internet-available information and consists of 36 items that are divided into 3 subsections. In addition, 5 guideline recommendations per analyzed condition were rephrased as questions and input to ChatGPT, and agreement between the guidelines and the AI answer was measured by 2 authors independently. All queries were repeated 3 times to measure the internal consistency of ChatGPT.

Results

Five conditions were identified (gallstone disease, pancreatitis, liver cirrhosis, pancreatic cancer, and hepatocellular carcinoma). The median EQIP score across all conditions was 16 (IQR 14.5-18) for the total of 36 items. Divided by subsection, median scores for content, identification, and structure data were 10 (IQR 9.5-12.5), 1 (IQR 1-1), and 4 (IQR 4-5), respectively. Agreement between guideline recommendations and answers provided by ChatGPT was 60% (15/25). Interrater agreement as measured by the Fleiss κ was 0.78 (P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceHealth PersonnelHumansReproducibility of ResultsInternetLanguage

Stichwörter der Autor:innen

ChatGPTEQIP toolartificial intelligencebilebiliarychatbotchatbotsconversational agentconversational agentsgallgallstonehepaticinternal medicineinternet informationliver

Autor:innen und Institutionen

  • Royal Free London NHS Foundation Trust, London, United Kingdom.
  • Clarunis - University Center for Gastrointestinal and Liver Diseases, Basel, Switzerland.
  • Departement Chirurgie, Kantonsspital Aarau, Aarau, Switzerland.
  • Organ Transplant Center of Excellence, King Faisal Specialist Hospital & Research Centre, Riyadh, Saudi Arabia.

ORCID-Profile: Harriet Louise Walker, Shahi Ghani, Christoph Kuemmerli, Christian Andreas Nebiker, Beat Peter Müller, Dimitri Aristotle Raptis, Sebastian Manuel Staubli

Zitieren

APA

Walker, H. L., Ghani, S., Kuemmerli, C., Nebiker, C. A., Müller, B. P., Raptis, D. A., & Staubli, S. M. (2023). Reliability of Medical Information Provided by ChatGPT: Assessment Against Clinical Guidelines and Patient Information Quality Instrument. Journal of Medical Internet Research, 25, e47479. https://doi.org/10.2196/47479

Vancouver

Walker HL, Ghani S, Kuemmerli C, Nebiker CA, Müller BP, Raptis DA, et al. Reliability of Medical Information Provided by ChatGPT: Assessment Against Clinical Guidelines and Patient Information Quality Instrument. Journal of Medical Internet Research. 2023;25:e47479. doi:10.2196/47479

BibTeX
@article{Walker2023,
  author = {Walker, Harriet Louise and Ghani, Shahi and Kuemmerli, Christoph and Nebiker, Christian Andreas and Müller, Beat Peter and Raptis, Dimitri Aristotle and Staubli, Sebastian Manuel},
  title = {Reliability of Medical Information Provided by ChatGPT: Assessment Against Clinical Guidelines and Patient Information Quality Instrument},
  journal = {Journal of Medical Internet Research},
  year = {2023},
  volume = {25},
  pages = {e47479},
  doi = {10.2196/47479}
}
Kostenlos starten →