Assessing the Efficacy of ChatGPT Versus Human Researchers in Identifying Relevant Studies on mHealth Interventions for Improving Medication Adherence in Patients With Ischemic Stroke When Conducting Systematic Reviews: Comparative Analysis
JMIR mHealth and uHealth · 2024-05-06 · 12 · e51526
DOI: 10.2196/51526 ↗At a glance
Journal article in JMIR mHealth and uHealth, published online on 2024-05-06. Written by Suebsarn Ruksakulpiwat and 6 co-authors. Institution: Department of Medical Nursing, Faculty of Nursing, Mahidol University, Bangkok, Thailand. Major MEDLINE topics: Medication Adherence, Telemedicine, Drug Monitoring. Cited 19 times so far (Crossref); 31 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
ChatGPT by OpenAI emerged as a potential tool for researchers, aiding in various aspects of research. One such application was the identification of relevant studies in systematic reviews. However, a comprehensive comparison of the efficacy of relevant study identification between human researchers and ChatGPT has not been conducted.
Objective
This study aims to compare the efficacy of ChatGPT and human researchers in identifying relevant studies on medication adherence improvement using mobile health interventions in patients with ischemic stroke during systematic reviews.
Methods
This study used the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. Four electronic databases, including CINAHL Plus with Full Text, Web of Science, PubMed, and MEDLINE, were searched to identify articles published from inception until 2023 using search terms based on MeSH (Medical Subject Headings) terms generated by human researchers versus ChatGPT. The authors independently screened the titles, abstracts, and full text of the studies identified through separate searches conducted by human researchers and ChatGPT. The comparison encompassed several aspects, including the ability to retrieve relevant studies, accuracy, efficiency, limitations, and challenges associated with each method.
Results
A total of 6 articles identified through search terms generated by human researchers were included in the final analysis, of which 4 (67%) reported improvements in medication adherence after the intervention. However, 33% (2/6) of the included studies did not clearly state whether medication adherence improved after the intervention. A total of 10 studies were included based on search terms generated by ChatGPT, of which 6 (60%) overlapped with studies identified by human researchers. Regarding the impact of mobile health interventions on medication adherence, most included studies (8/10, 80%) based on search terms generated by ChatGPT reported improvements in medication adherence after the intervention. However, 20% (2/10) of the studies did not clearly state whether medication adherence improved after the intervention. The precision in accurately identifying relevant studies was higher in human researchers (0.86) than in ChatGPT (0.77). This is consistent with the percentage of relevance, where human researchers (9.8%) demonstrated a higher percentage of relevance than ChatGPT (3%). However, when considering the time required for both humans and ChatGPT to identify relevant studies, ChatGPT substantially outperformed human researchers as it took less time to identify relevant studies.
Conclusions
Our comparative analysis highlighted the strengths and limitations of both approaches. Ultimately, the choice between human researchers and ChatGPT depends on the specific requirements and objectives of each review, but the collaborative synergy of both approaches holds the potential to advance evidence-based research and decision-making in the health care field.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Medical Nursing, Faculty of Nursing, Mahidol University, Bangkok, Thailand.
- Jack, Joseph and Morton Mandel School of Applied Social Sciences, Case Western Reserve University, Cleveland, OH, United States.
- College of Art and Science, Department of Anthropology, Case Western Reserve University, Cleveland, OH, United States.
- School of Medicine, Case Western Reserve University, Cleveland, OH, United States.
- Frances Payne Bolton School of Nursing, Case Western Reserve University, Cleveland, OH, United States.
ORCID profiles: Suebsarn Ruksakulpiwat, Lalipat Phianhasin, Chitchanok Benjasirisan, Kedong Ding, Anuoluwapo Ajibade, Ayanesh Kumar, Cassie Stewart
Cite
APA
Ruksakulpiwat, S., Phianhasin, L., Benjasirisan, C., Ding, K., Ajibade, A., Kumar, A., & Stewart, C. (2024). Assessing the Efficacy of ChatGPT Versus Human Researchers in Identifying Relevant Studies on mHealth Interventions for Improving Medication Adherence in Patients With Ischemic Stroke When Conducting Systematic Reviews: Comparative Analysis. JMIR mHealth and uHealth, 12, e51526. https://doi.org/10.2196/51526
Vancouver
Ruksakulpiwat S, Phianhasin L, Benjasirisan C, Ding K, Ajibade A, Kumar A, et al. Assessing the Efficacy of ChatGPT Versus Human Researchers in Identifying Relevant Studies on mHealth Interventions for Improving Medication Adherence in Patients With Ischemic Stroke When Conducting Systematic Reviews: Comparative Analysis. JMIR mHealth and uHealth. 2024;12:e51526. doi:10.2196/51526
BibTeX
@article{Ruksakulpiwat2024,
author = {Ruksakulpiwat, Suebsarn and Phianhasin, Lalipat and Benjasirisan, Chitchanok and Ding, Kedong and Ajibade, Anuoluwapo and Kumar, Ayanesh and Stewart, Cassie},
title = {Assessing the Efficacy of ChatGPT Versus Human Researchers in Identifying Relevant Studies on mHealth Interventions for Improving Medication Adherence in Patients With Ischemic Stroke When Conducting Systematic Reviews: Comparative Analysis},
journal = {JMIR mHealth and uHealth},
year = {2024},
volume = {12},
pages = {e51526},
doi = {10.2196/51526}
}
EUVIMED