EUVIMEDEUVIMED entdecken
Qualitative StudieOpen Access

Evaluating the Capabilities of Generative AI Tools in Understanding Medical Papers: Qualitative Study

Seyma Handan Akyon iD, Fatih Cagatay Akyon iD, Ahmet Sefa Camyar iD, Fatih Hızlı iD, Talha Sari iD, Şamil Hızlı iD

JMIR Medical Informatics · 04.09.2024 · 12 · e59258

DOI: 10.2196/59258 ↗

Kurz gefasst

Qualitative Studie in der Zeitschrift JMIR Medical Informatics, online erschienen am 04.09.2024. Verfasst von Seyma Handan Akyon und 5 weiteren Autor:innen. Beteiligte Einrichtung: Golpazari Family Health Center, Bilecik, Turkey. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Reading medical papers is a challenging and time-consuming task for doctors, especially when the papers are long and complex. A tool that can help doctors efficiently process and understand medical papers is needed.

Objective

This study aims to critically assess and compare the comprehension capabilities of large language models (LLMs) in accurately and efficiently understanding medical research papers using the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) checklist, which provides a standardized framework for evaluating key elements of observational study.

Methods

The study is a methodological type of research. The study aims to evaluate the understanding capabilities of new generative artificial intelligence tools in medical papers. A novel benchmark pipeline processed 50 medical research papers from PubMed, comparing the answers of 6 LLMs (GPT-3.5-Turbo, GPT-4-0613, GPT-4-1106, PaLM 2, Claude v1, and Gemini Pro) to the benchmark established by expert medical professors. Fifteen questions, derived from the STROBE checklist, assessed LLMs’ understanding of different sections of a research paper.

Results

LLMs exhibited varying performance, with GPT-3.5-Turbo achieving the highest percentage of correct answers (n=3916, 66.9%), followed by GPT-4-1106 (n=3837, 65.6%), PaLM 2 (n=3632, 62.1%), Claude v1 (n=2887, 58.3%), Gemini Pro (n=2878, 49.2%), and GPT-4-0613 (n=2580, 44.1%). Statistical analysis revealed statistically significant differences between LLMs (P

Schlagwörter

Stichwörter der Autor:innen

AIChatGPTGPTLLMLLMsSTROBEStrengthening the Reporting of Observational Studies in Epidemiologyansweranswersartificial intelligencecomprehensiongenerativehealth carelanguage modellanguage models

Autor:innen und Institutionen

  • Golpazari Family Health Center, Bilecik, Turkey.
  • SafeVideo AI, San Francisco, CA, United States.
  • Graduate School of Informatics, Middle East Technical University, Ankara, Turkey.
  • Department of Internal Medicine, Ankara Etlik City Hospital, Ankara, Turkey.
  • Faculty of Medicine, Ankara Yildirim Beyazit University, Ankara, Turkey.
  • Department of Computer Science, Istanbul Technical University, Istanbul, Turkey.

ORCID-Profile: Seyma Handan Akyon, Fatih Cagatay Akyon, Ahmet Sefa Camyar, Fatih Hızlı, Talha Sari, Şamil Hızlı

Zitieren

APA

Akyon, S. H., Akyon, F. C., Camyar, A. S., Hızlı, F., Sari, T., & Hızlı, Ş. (2024). Evaluating the Capabilities of Generative AI Tools in Understanding Medical Papers: Qualitative Study. JMIR Medical Informatics, 12, e59258. https://doi.org/10.2196/59258

Vancouver

Akyon SH, Akyon FC, Camyar AS, Hızlı F, Sari T, Hızlı Ş. Evaluating the Capabilities of Generative AI Tools in Understanding Medical Papers: Qualitative Study. JMIR Medical Informatics. 2024;12:e59258. doi:10.2196/59258

BibTeX
@article{Akyon2024,
  author = {Akyon, Seyma Handan and Akyon, Fatih Cagatay and Camyar, Ahmet Sefa and Hızlı, Fatih and Sari, Talha and Hızlı, Şamil},
  title = {Evaluating the Capabilities of Generative AI Tools in Understanding Medical Papers: Qualitative Study},
  journal = {JMIR Medical Informatics},
  year = {2024},
  volume = {12},
  pages = {e59258},
  doi = {10.2196/59258}
}
Kostenlos starten →