EUVIMEDEUVIMED entdecken
Open Access

The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation

Yong Nam Gwon iD, Jae Heon Kim iD, Hyun Soo Chung iD, Eun Jee Jung iD, Joey Chun iD, Serin Lee iD, Sung Ryul Shim iD

JMIR Medical Informatics · 14.05.2024 · 12 · e51187-e51187

DOI: 10.2196/51187 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Medical Informatics, online erschienen am 14.05.2024. Verfasst von Yong Nam Gwon und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Urology, Soonchunhyang University College of Medicine, Soonchunhyang University Seoul Hospital, Seoul, Republic of Korea. Die Arbeit wurde bisher 52-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

A large language model is a type of artificial intelligence (AI) model that opens up great possibilities for health care practice, research, and education, although scholars have emphasized the need to proactively address the issue of unvalidated and inaccurate information regarding its use. One of the best-known large language models is ChatGPT (OpenAI). It is believed to be of great help to medical research, as it facilitates more efficient data set analysis, code generation, and literature review, allowing researchers to focus on experimental design as well as drug discovery and development.

Objective

This study aims to explore the potential of ChatGPT as a real-time literature search tool for systematic reviews and clinical decision support systems, to enhance their efficiency and accuracy in health care settings.

Methods

The search results of a published systematic review by human experts on the treatment of Peyronie disease were selected as a benchmark, and the literature search formula of the study was applied to ChatGPT and Microsoft Bing AI as a comparison to human researchers. Peyronie disease typically presents with discomfort, curvature, or deformity of the penis in association with palpable plaques and erectile dysfunction. To evaluate the quality of individual studies derived from AI answers, we created a structured rating system based on bibliographic information related to the publications. We classified its answers into 4 grades if the title existed: A, B, C, and F. No grade was given for a fake title or no answer.

Results

From ChatGPT, 7 (0.5%) out of 1287 identified studies were directly relevant, whereas Bing AI resulted in 19 (40%) relevant studies out of 48, compared to the human benchmark of 24 studies. In the qualitative evaluation, ChatGPT had 7 grade A, 18 grade B, 167 grade C, and 211 grade F studies, and Bing AI had 19 grade A and 28 grade C studies.

Conclusions

This is the first study to compare AI and conventional human systematic review methods as a real-time literature collection tool for evidence-based medicine. The results suggest that the use of ChatGPT as a tool for real-time evidence generation is not yet accurate and feasible. Therefore, researchers should be cautious about using such AI. The limitations of this study using the generative pre-trained transformer model are that the search for research topics was not diverse and that it did not prevent the hallucination of generative AI. However, this study will serve as a standard for future studies by providing an index to verify the reliability and consistency of generative AI from a user’s point of view. If the reliability and consistency of AI literature search services are verified, then the use of these technologies will help medical research greatly.

Schlagwörter

Stichwörter der Autor:innen

ChatGPTartificial intelligenceclinical decision support systemdecision supporteducationevidence-based medicinelanguage modelsearch enginesupportsystematic reviewtooltreatment

Autor:innen und Institutionen

  • Department of Urology, Soonchunhyang University College of Medicine, Soonchunhyang University Seoul Hospital, Seoul, Republic of Korea.
  • College of Medicine, Soonchunhyang University, Cheonan, Republic of Korea.
  • Cranbrook Kingswood Upper School, Bloomfield Hills, MI, United States.
  • Department of Biochemistry, Case Western Reserve University, Cleveland, OH, United States.
  • Department of Biomedical Informatics, Konyang University College of Medicine, Daejeon, Republic of Korea.
  • Konyang Medical Data Research Group-KYMERA, Konyang University Hospital, Daejeon, Republic of Korea.

ORCID-Profile: Yong Nam Gwon, Jae Heon Kim, Hyun Soo Chung, Eun Jee Jung, Joey Chun, Serin Lee, Sung Ryul Shim

Zitieren

APA

Gwon, Y. N., Kim, J. H., Chung, H. S., Jung, E. J., Chun, J., Lee, S., & Shim, S. R. (2024). The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation. JMIR Medical Informatics, 12, e51187-e51187. https://doi.org/10.2196/51187

Vancouver

Gwon YN, Kim JH, Chung HS, Jung EJ, Chun J, Lee S, et al. The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation. JMIR Medical Informatics. 2024;12:e51187-e51187. doi:10.2196/51187

BibTeX
@article{Gwon2024,
  author = {Gwon, Yong Nam and Kim, Jae Heon and Chung, Hyun Soo and Jung, Eun Jee and Chun, Joey and Lee, Serin and Shim, Sung Ryul},
  title = {The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation},
  journal = {JMIR Medical Informatics},
  year = {2024},
  volume = {12},
  pages = {e51187--e51187},
  doi = {10.2196/51187}
}
Kostenlos starten →