Comparing the Accuracy of Two Generated Large Language Models in Identifying Health-Related Rumors or Misconceptions and the Applicability in Health Science Popularization: Proof-of-Concept Study
JMIR Formative Research · 2024
DOI: 10.2196/63188 ↗At a glance
Vergleichsstudie in JMIR Formative Research, 2024. Written by Yuan Luo and 6 co-authors. Institution: School of Nursing, Capital Medical University, 10 Xitoutiao, Youanmen Wai, Fengtai District, Beijing, 100069, China, 86 10839117. Major MEDLINE topics: Social Media. Cited 7 times so far (Crossref); 45 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Health-related rumors and misconceptions are spreading at an alarming rate, fueled by the rapid development of the internet and the exponential growth of social media platforms. This phenomenon has become a pressing global concern, as the dissemination of false information can have severe consequences, including widespread panic, social instability, and even public health crises.
Objective
The aim of the study is to compare the accuracy of rumor identification and the effectiveness of health science popularization between 2 generated large language models in Chinese (GPT-4 by OpenAI and Enhanced Representation through Knowledge Integration Bot [ERNIE Bot] 4.0 by Baidu).
Methods
In total, 20 health rumors and misconceptions, along with 10 health truths, were randomly inputted into GPT-4 and ERNIE Bot 4.0. We prompted them to determine whether the statements were rumors or misconceptions and provide explanations for their judgment. Further, we asked them to generate a health science popularization essay. We evaluated the outcomes in terms of accuracy, effectiveness, readability, and applicability. Accuracy was assessed by the rate of correctly identifying health-related rumors, misconceptions, and truths. Effectiveness was determined by the accuracy of the generated explanation, which was assessed collaboratively by 2 research team members with a PhD in nursing. Readability was calculated by the readability formula of Chinese health education materials. Applicability was evaluated by the Chinese Suitability Assessment of Materials.
Results
GPT-4 and ERNIE Bot 4.0 correctly identified all health rumors and misconceptions (100% accuracy rate). For truths, the accuracy rate was 70% (7/10) and 100% (10/10), respectively. Both mostly provided widely recognized viewpoints without obvious errors. The average readability score for the health essays was 2.92 (SD 0.85) for GPT-4 and 3.02 (SD 0.84) for ERNIE Bot 4.0 ( P =.65). For applicability, except for the content and cultural appropriateness category, significant differences were observed in the total score and scores in other dimensions between them ( P
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- School of Nursing, Capital Medical University, 10 Xitoutiao, Youanmen Wai, Fengtai District, Beijing, 100069, China, 86 10839117.
- School of Nursing, Johns Hopkins University, Baltimore, MD, United States.
ORCID profiles: Yuan Luo, Yiqun Miao, Yuhan Zhao, Jiawei Li, Yuling Chen, Yuexue Yue, Ying Wu
Cite
APA
Luo, Y., Miao, Y., Zhao, Y., Li, J., Chen, Y., Yue, Y., & Wu, Y. (2024). Comparing the Accuracy of Two Generated Large Language Models in Identifying Health-Related Rumors or Misconceptions and the Applicability in Health Science Popularization: Proof-of-Concept Study. JMIR Formative Research. https://doi.org/10.2196/63188
Vancouver
Luo Y, Miao Y, Zhao Y, Li J, Chen Y, Yue Y, et al. Comparing the Accuracy of Two Generated Large Language Models in Identifying Health-Related Rumors or Misconceptions and the Applicability in Health Science Popularization: Proof-of-Concept Study. JMIR Formative Research. 2024. doi:10.2196/63188
BibTeX
@article{Luo2024,
author = {Luo, Yuan and Miao, Yiqun and Zhao, Yuhan and Li, Jiawei and Chen, Yuling and Yue, Yuexue and Wu, Ying},
title = {Comparing the Accuracy of Two Generated Large Language Models in Identifying Health-Related Rumors or Misconceptions and the Applicability in Health Science Popularization: Proof-of-Concept Study},
journal = {JMIR Formative Research},
year = {2024},
doi = {10.2196/63188}
}
EUVIMED