Evaluating the Capabilities of Generative AI Tools in Understanding Medical Papers: Qualitative Study
JMIR Medical Informatics · 2024-09-04 · 12 · e59258
DOI: 10.2196/59258 ↗At a glance
Qualitative Studie in JMIR Medical Informatics, published online on 2024-09-04. Written by Seyma Handan Akyon and 5 co-authors. Institution: Golpazari Family Health Center, Bilecik, Turkey. Cited 14 times so far (Crossref); 34 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Reading medical papers is a challenging and time-consuming task for doctors, especially when the papers are long and complex. A tool that can help doctors efficiently process and understand medical papers is needed.
Objective
This study aims to critically assess and compare the comprehension capabilities of large language models (LLMs) in accurately and efficiently understanding medical research papers using the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) checklist, which provides a standardized framework for evaluating key elements of observational study.
Methods
The study is a methodological type of research. The study aims to evaluate the understanding capabilities of new generative artificial intelligence tools in medical papers. A novel benchmark pipeline processed 50 medical research papers from PubMed, comparing the answers of 6 LLMs (GPT-3.5-Turbo, GPT-4-0613, GPT-4-1106, PaLM 2, Claude v1, and Gemini Pro) to the benchmark established by expert medical professors. Fifteen questions, derived from the STROBE checklist, assessed LLMs’ understanding of different sections of a research paper.
Results
LLMs exhibited varying performance, with GPT-3.5-Turbo achieving the highest percentage of correct answers (n=3916, 66.9%), followed by GPT-4-1106 (n=3837, 65.6%), PaLM 2 (n=3632, 62.1%), Claude v1 (n=2887, 58.3%), Gemini Pro (n=2878, 49.2%), and GPT-4-0613 (n=2580, 44.1%). Statistical analysis revealed statistically significant differences between LLMs (P
Subjects
Author keywords
Authors and institutions
- Golpazari Family Health Center, Bilecik, Turkey.
- SafeVideo AI, San Francisco, CA, United States.
- Graduate School of Informatics, Middle East Technical University, Ankara, Turkey.
- Department of Internal Medicine, Ankara Etlik City Hospital, Ankara, Turkey.
- Faculty of Medicine, Ankara Yildirim Beyazit University, Ankara, Turkey.
- Department of Computer Science, Istanbul Technical University, Istanbul, Turkey.
ORCID profiles: Seyma Handan Akyon, Fatih Cagatay Akyon, Ahmet Sefa Camyar, Fatih Hızlı, Talha Sari, Şamil Hızlı
Cite
APA
Akyon, S. H., Akyon, F. C., Camyar, A. S., Hızlı, F., Sari, T., & Hızlı, Ş. (2024). Evaluating the Capabilities of Generative AI Tools in Understanding Medical Papers: Qualitative Study. JMIR Medical Informatics, 12, e59258. https://doi.org/10.2196/59258
Vancouver
Akyon SH, Akyon FC, Camyar AS, Hızlı F, Sari T, Hızlı Ş. Evaluating the Capabilities of Generative AI Tools in Understanding Medical Papers: Qualitative Study. JMIR Medical Informatics. 2024;12:e59258. doi:10.2196/59258
BibTeX
@article{Akyon2024,
author = {Akyon, Seyma Handan and Akyon, Fatih Cagatay and Camyar, Ahmet Sefa and Hızlı, Fatih and Sari, Talha and Hızlı, Şamil},
title = {Evaluating the Capabilities of Generative AI Tools in Understanding Medical Papers: Qualitative Study},
journal = {JMIR Medical Informatics},
year = {2024},
volume = {12},
pages = {e59258},
doi = {10.2196/59258}
}
EUVIMED