EUVIMEDSkopri EUVIMED
Qualitative StudieOpen access

Evaluating the Capabilities of Generative AI Tools in Understanding Medical Papers: Qualitative Study

Seyma Handan Akyon iD, Fatih Cagatay Akyon iD, Ahmet Sefa Camyar iD, Fatih Hızlı iD, Talha Sari iD, Şamil Hızlı iD

JMIR Medical Informatics · 2024-09-04 · 12 · e59258

DOI: 10.2196/59258 ↗

At a glance

Qualitative Studie in JMIR Medical Informatics, published online on 2024-09-04. Written by Seyma Handan Akyon and 5 co-authors. Institution: Golpazari Family Health Center, Bilecik, Turkey. Cited 14 times so far (Crossref); 34 references. The full text is freely available in PubMed Central.

AstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Background

Reading medical papers is a challenging and time-consuming task for doctors, especially when the papers are long and complex. A tool that can help doctors efficiently process and understand medical papers is needed.

Objective

This study aims to critically assess and compare the comprehension capabilities of large language models (LLMs) in accurately and efficiently understanding medical research papers using the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) checklist, which provides a standardized framework for evaluating key elements of observational study.

Methods

The study is a methodological type of research. The study aims to evaluate the understanding capabilities of new generative artificial intelligence tools in medical papers. A novel benchmark pipeline processed 50 medical research papers from PubMed, comparing the answers of 6 LLMs (GPT-3.5-Turbo, GPT-4-0613, GPT-4-1106, PaLM 2, Claude v1, and Gemini Pro) to the benchmark established by expert medical professors. Fifteen questions, derived from the STROBE checklist, assessed LLMs’ understanding of different sections of a research paper.

Results

LLMs exhibited varying performance, with GPT-3.5-Turbo achieving the highest percentage of correct answers (n=3916, 66.9%), followed by GPT-4-1106 (n=3837, 65.6%), PaLM 2 (n=3632, 62.1%), Claude v1 (n=2887, 58.3%), Gemini Pro (n=2878, 49.2%), and GPT-4-0613 (n=2580, 44.1%). Statistical analysis revealed statistically significant differences between LLMs (P

Subjects

Author keywords

AIChatGPTGPTLLMLLMsSTROBEStrengthening the Reporting of Observational Studies in Epidemiologyansweranswersartificial intelligencecomprehensiongenerativehealth carelanguage modellanguage models

Authors and institutions

  • Golpazari Family Health Center, Bilecik, Turkey.
  • SafeVideo AI, San Francisco, CA, United States.
  • Graduate School of Informatics, Middle East Technical University, Ankara, Turkey.
  • Department of Internal Medicine, Ankara Etlik City Hospital, Ankara, Turkey.
  • Faculty of Medicine, Ankara Yildirim Beyazit University, Ankara, Turkey.
  • Department of Computer Science, Istanbul Technical University, Istanbul, Turkey.

ORCID profiles: Seyma Handan Akyon, Fatih Cagatay Akyon, Ahmet Sefa Camyar, Fatih Hızlı, Talha Sari, Şamil Hızlı

Cite

APA

Akyon, S. H., Akyon, F. C., Camyar, A. S., Hızlı, F., Sari, T., & Hızlı, Ş. (2024). Evaluating the Capabilities of Generative AI Tools in Understanding Medical Papers: Qualitative Study. JMIR Medical Informatics, 12, e59258. https://doi.org/10.2196/59258

Vancouver

Akyon SH, Akyon FC, Camyar AS, Hızlı F, Sari T, Hızlı Ş. Evaluating the Capabilities of Generative AI Tools in Understanding Medical Papers: Qualitative Study. JMIR Medical Informatics. 2024;12:e59258. doi:10.2196/59258

BibTeX
@article{Akyon2024,
  author = {Akyon, Seyma Handan and Akyon, Fatih Cagatay and Camyar, Ahmet Sefa and Hızlı, Fatih and Sari, Talha and Hızlı, Şamil},
  title = {Evaluating the Capabilities of Generative AI Tools in Understanding Medical Papers: Qualitative Study},
  journal = {JMIR Medical Informatics},
  year = {2024},
  volume = {12},
  pages = {e59258},
  doi = {10.2196/59258}
}
Start for free →