EUVIMEDSkopri EUVIMED
Open access

Assessing the Efficacy of ChatGPT Versus Human Researchers in Identifying Relevant Studies on mHealth Interventions for Improving Medication Adherence in Patients With Ischemic Stroke When Conducting Systematic Reviews: Comparative Analysis

Suebsarn Ruksakulpiwat iD, Lalipat Phianhasin iD, Chitchanok Benjasirisan iD, Kedong Ding iD, Anuoluwapo Ajibade iD, Ayanesh Kumar iD, Cassie Stewart iD

JMIR mHealth and uHealth · 2024-05-06 · 12 · e51526

DOI: 10.2196/51526 ↗

At a glance

Journal article in JMIR mHealth and uHealth, published online on 2024-05-06. Written by Suebsarn Ruksakulpiwat and 6 co-authors. Institution: Department of Medical Nursing, Faculty of Nursing, Mahidol University, Bangkok, Thailand. Major MEDLINE topics: Medication Adherence, Telemedicine, Drug Monitoring. Cited 19 times so far (Crossref); 31 references. The full text is freely available in PubMed Central.

AstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Background

ChatGPT by OpenAI emerged as a potential tool for researchers, aiding in various aspects of research. One such application was the identification of relevant studies in systematic reviews. However, a comprehensive comparison of the efficacy of relevant study identification between human researchers and ChatGPT has not been conducted.

Objective

This study aims to compare the efficacy of ChatGPT and human researchers in identifying relevant studies on medication adherence improvement using mobile health interventions in patients with ischemic stroke during systematic reviews.

Methods

This study used the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. Four electronic databases, including CINAHL Plus with Full Text, Web of Science, PubMed, and MEDLINE, were searched to identify articles published from inception until 2023 using search terms based on MeSH (Medical Subject Headings) terms generated by human researchers versus ChatGPT. The authors independently screened the titles, abstracts, and full text of the studies identified through separate searches conducted by human researchers and ChatGPT. The comparison encompassed several aspects, including the ability to retrieve relevant studies, accuracy, efficiency, limitations, and challenges associated with each method.

Results

A total of 6 articles identified through search terms generated by human researchers were included in the final analysis, of which 4 (67%) reported improvements in medication adherence after the intervention. However, 33% (2/6) of the included studies did not clearly state whether medication adherence improved after the intervention. A total of 10 studies were included based on search terms generated by ChatGPT, of which 6 (60%) overlapped with studies identified by human researchers. Regarding the impact of mobile health interventions on medication adherence, most included studies (8/10, 80%) based on search terms generated by ChatGPT reported improvements in medication adherence after the intervention. However, 20% (2/10) of the studies did not clearly state whether medication adherence improved after the intervention. The precision in accurately identifying relevant studies was higher in human researchers (0.86) than in ChatGPT (0.77). This is consistent with the percentage of relevance, where human researchers (9.8%) demonstrated a higher percentage of relevance than ChatGPT (3%). However, when considering the time required for both humans and ChatGPT to identify relevant studies, ChatGPT substantially outperformed human researchers as it took less time to identify relevant studies.

Conclusions

Our comparative analysis highlighted the strengths and limitations of both approaches. Ultimately, the choice between human researchers and ChatGPT depends on the specific requirements and objectives of each review, but the collaborative synergy of both approaches holds the potential to advance evidence-based research and decision-making in the health care field.

Subjects

MeSH terms (MEDLINE)

Medication AdherenceTelemedicineDrug MonitoringHumansIschemic StrokeSystematic Reviews as TopicResearch Personnel

Author keywords

ChatGPTischemic strokemHealthmedication adherencemobile healthmobile phonesystematic reviews

Authors and institutions

  • Department of Medical Nursing, Faculty of Nursing, Mahidol University, Bangkok, Thailand.
  • Jack, Joseph and Morton Mandel School of Applied Social Sciences, Case Western Reserve University, Cleveland, OH, United States.
  • College of Art and Science, Department of Anthropology, Case Western Reserve University, Cleveland, OH, United States.
  • School of Medicine, Case Western Reserve University, Cleveland, OH, United States.
  • Frances Payne Bolton School of Nursing, Case Western Reserve University, Cleveland, OH, United States.

ORCID profiles: Suebsarn Ruksakulpiwat, Lalipat Phianhasin, Chitchanok Benjasirisan, Kedong Ding, Anuoluwapo Ajibade, Ayanesh Kumar, Cassie Stewart

Cite

APA

Ruksakulpiwat, S., Phianhasin, L., Benjasirisan, C., Ding, K., Ajibade, A., Kumar, A., & Stewart, C. (2024). Assessing the Efficacy of ChatGPT Versus Human Researchers in Identifying Relevant Studies on mHealth Interventions for Improving Medication Adherence in Patients With Ischemic Stroke When Conducting Systematic Reviews: Comparative Analysis. JMIR mHealth and uHealth, 12, e51526. https://doi.org/10.2196/51526

Vancouver

Ruksakulpiwat S, Phianhasin L, Benjasirisan C, Ding K, Ajibade A, Kumar A, et al. Assessing the Efficacy of ChatGPT Versus Human Researchers in Identifying Relevant Studies on mHealth Interventions for Improving Medication Adherence in Patients With Ischemic Stroke When Conducting Systematic Reviews: Comparative Analysis. JMIR mHealth and uHealth. 2024;12:e51526. doi:10.2196/51526

BibTeX
@article{Ruksakulpiwat2024,
  author = {Ruksakulpiwat, Suebsarn and Phianhasin, Lalipat and Benjasirisan, Chitchanok and Ding, Kedong and Ajibade, Anuoluwapo and Kumar, Ayanesh and Stewart, Cassie},
  title = {Assessing the Efficacy of ChatGPT Versus Human Researchers in Identifying Relevant Studies on mHealth Interventions for Improving Medication Adherence in Patients With Ischemic Stroke When Conducting Systematic Reviews: Comparative Analysis},
  journal = {JMIR mHealth and uHealth},
  year = {2024},
  volume = {12},
  pages = {e51526},
  doi = {10.2196/51526}
}
Start for free →