EUVIMEDPoznaj EUVIMED
Open access

The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation

Yong Nam Gwon iD, Jae Heon Kim iD, Hyun Soo Chung iD, Eun Jee Jung iD, Joey Chun iD, Serin Lee iD, Sung Ryul Shim iD

JMIR Medical Informatics · 2024-05-14 · 12 · e51187-e51187

DOI: 10.2196/51187 ↗

At a glance

Journal article in JMIR Medical Informatics, published online on 2024-05-14. Written by Yong Nam Gwon and 6 co-authors. Institution: Department of Urology, Soonchunhyang University College of Medicine, Soonchunhyang University Seoul Hospital, Seoul, Republic of Korea. Cited 52 times so far (Crossref); 32 references. The full text is freely available in PubMed Central.

AbstraktSubjectsDane bibliograficzneCiteRelated studies

Abstract (original language)

Background

A large language model is a type of artificial intelligence (AI) model that opens up great possibilities for health care practice, research, and education, although scholars have emphasized the need to proactively address the issue of unvalidated and inaccurate information regarding its use. One of the best-known large language models is ChatGPT (OpenAI). It is believed to be of great help to medical research, as it facilitates more efficient data set analysis, code generation, and literature review, allowing researchers to focus on experimental design as well as drug discovery and development.

Objective

This study aims to explore the potential of ChatGPT as a real-time literature search tool for systematic reviews and clinical decision support systems, to enhance their efficiency and accuracy in health care settings.

Methods

The search results of a published systematic review by human experts on the treatment of Peyronie disease were selected as a benchmark, and the literature search formula of the study was applied to ChatGPT and Microsoft Bing AI as a comparison to human researchers. Peyronie disease typically presents with discomfort, curvature, or deformity of the penis in association with palpable plaques and erectile dysfunction. To evaluate the quality of individual studies derived from AI answers, we created a structured rating system based on bibliographic information related to the publications. We classified its answers into 4 grades if the title existed: A, B, C, and F. No grade was given for a fake title or no answer.

Results

From ChatGPT, 7 (0.5%) out of 1287 identified studies were directly relevant, whereas Bing AI resulted in 19 (40%) relevant studies out of 48, compared to the human benchmark of 24 studies. In the qualitative evaluation, ChatGPT had 7 grade A, 18 grade B, 167 grade C, and 211 grade F studies, and Bing AI had 19 grade A and 28 grade C studies.

Conclusions

This is the first study to compare AI and conventional human systematic review methods as a real-time literature collection tool for evidence-based medicine. The results suggest that the use of ChatGPT as a tool for real-time evidence generation is not yet accurate and feasible. Therefore, researchers should be cautious about using such AI. The limitations of this study using the generative pre-trained transformer model are that the search for research topics was not diverse and that it did not prevent the hallucination of generative AI. However, this study will serve as a standard for future studies by providing an index to verify the reliability and consistency of generative AI from a user’s point of view. If the reliability and consistency of AI literature search services are verified, then the use of these technologies will help medical research greatly.

Subjects

Author keywords

ChatGPTartificial intelligenceclinical decision support systemdecision supporteducationevidence-based medicinelanguage modelsearch enginesupportsystematic reviewtooltreatment

Authors and institutions

  • Department of Urology, Soonchunhyang University College of Medicine, Soonchunhyang University Seoul Hospital, Seoul, Republic of Korea.
  • College of Medicine, Soonchunhyang University, Cheonan, Republic of Korea.
  • Cranbrook Kingswood Upper School, Bloomfield Hills, MI, United States.
  • Department of Biochemistry, Case Western Reserve University, Cleveland, OH, United States.
  • Department of Biomedical Informatics, Konyang University College of Medicine, Daejeon, Republic of Korea.
  • Konyang Medical Data Research Group-KYMERA, Konyang University Hospital, Daejeon, Republic of Korea.

ORCID profiles: Yong Nam Gwon, Jae Heon Kim, Hyun Soo Chung, Eun Jee Jung, Joey Chun, Serin Lee, Sung Ryul Shim

Cite

APA

Gwon, Y. N., Kim, J. H., Chung, H. S., Jung, E. J., Chun, J., Lee, S., & Shim, S. R. (2024). The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation. JMIR Medical Informatics, 12, e51187-e51187. https://doi.org/10.2196/51187

Vancouver

Gwon YN, Kim JH, Chung HS, Jung EJ, Chun J, Lee S, et al. The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation. JMIR Medical Informatics. 2024;12:e51187-e51187. doi:10.2196/51187

BibTeX
@article{Gwon2024,
  author = {Gwon, Yong Nam and Kim, Jae Heon and Chung, Hyun Soo and Jung, Eun Jee and Chun, Joey and Lee, Serin and Shim, Sung Ryul},
  title = {The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation},
  journal = {JMIR Medical Informatics},
  year = {2024},
  volume = {12},
  pages = {e51187--e51187},
  doi = {10.2196/51187}
}
Start for free →