EUVIMEDOpdag EUVIMED
VergleichsstudieOpen access

Comparing the Accuracy of Two Generated Large Language Models in Identifying Health-Related Rumors or Misconceptions and the Applicability in Health Science Popularization: Proof-of-Concept Study

Yuan Luo iD, Yiqun Miao iD, Yuhan Zhao iD, Jiawei Li iD, Yuling Chen iD, Yuexue Yue iD, Ying Wu iD

JMIR Formative Research · 2024

DOI: 10.2196/63188 ↗

At a glance

Vergleichsstudie in JMIR Formative Research, 2024. Written by Yuan Luo and 6 co-authors. Institution: School of Nursing, Capital Medical University, 10 Xitoutiao, Youanmen Wai, Fengtai District, Beijing, 100069, China, 86 10839117. Major MEDLINE topics: Social Media. Cited 7 times so far (Crossref); 45 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Health-related rumors and misconceptions are spreading at an alarming rate, fueled by the rapid development of the internet and the exponential growth of social media platforms. This phenomenon has become a pressing global concern, as the dissemination of false information can have severe consequences, including widespread panic, social instability, and even public health crises.

Objective

The aim of the study is to compare the accuracy of rumor identification and the effectiveness of health science popularization between 2 generated large language models in Chinese (GPT-4 by OpenAI and Enhanced Representation through Knowledge Integration Bot [ERNIE Bot] 4.0 by Baidu).

Methods

In total, 20 health rumors and misconceptions, along with 10 health truths, were randomly inputted into GPT-4 and ERNIE Bot 4.0. We prompted them to determine whether the statements were rumors or misconceptions and provide explanations for their judgment. Further, we asked them to generate a health science popularization essay. We evaluated the outcomes in terms of accuracy, effectiveness, readability, and applicability. Accuracy was assessed by the rate of correctly identifying health-related rumors, misconceptions, and truths. Effectiveness was determined by the accuracy of the generated explanation, which was assessed collaboratively by 2 research team members with a PhD in nursing. Readability was calculated by the readability formula of Chinese health education materials. Applicability was evaluated by the Chinese Suitability Assessment of Materials.

Results

GPT-4 and ERNIE Bot 4.0 correctly identified all health rumors and misconceptions (100% accuracy rate). For truths, the accuracy rate was 70% (7/10) and 100% (10/10), respectively. Both mostly provided widely recognized viewpoints without obvious errors. The average readability score for the health essays was 2.92 (SD 0.85) for GPT-4 and 3.02 (SD 0.84) for ERNIE Bot 4.0 ( P =.65). For applicability, except for the content and cultural appropriateness category, significant differences were observed in the total score and scores in other dimensions between them ( P

Subjects

MeSH terms (MEDLINE)

Social MediaHumansProof of Concept StudyChinaLanguage

Author keywords

LLMaccuracyapplicabilityeducationeffectivenesshealth educationhealth relatedhealth sciencehealth science popularizationlarge language modelmisconceptionproof of conceptrumor

Authors and institutions

  • School of Nursing, Capital Medical University, 10 Xitoutiao, Youanmen Wai, Fengtai District, Beijing, 100069, China, 86 10839117.
  • School of Nursing, Johns Hopkins University, Baltimore, MD, United States.

ORCID profiles: Yuan Luo, Yiqun Miao, Yuhan Zhao, Jiawei Li, Yuling Chen, Yuexue Yue, Ying Wu

Cite

APA

Luo, Y., Miao, Y., Zhao, Y., Li, J., Chen, Y., Yue, Y., & Wu, Y. (2024). Comparing the Accuracy of Two Generated Large Language Models in Identifying Health-Related Rumors or Misconceptions and the Applicability in Health Science Popularization: Proof-of-Concept Study. JMIR Formative Research. https://doi.org/10.2196/63188

Vancouver

Luo Y, Miao Y, Zhao Y, Li J, Chen Y, Yue Y, et al. Comparing the Accuracy of Two Generated Large Language Models in Identifying Health-Related Rumors or Misconceptions and the Applicability in Health Science Popularization: Proof-of-Concept Study. JMIR Formative Research. 2024. doi:10.2196/63188

BibTeX
@article{Luo2024,
  author = {Luo, Yuan and Miao, Yiqun and Zhao, Yuhan and Li, Jiawei and Chen, Yuling and Yue, Yuexue and Wu, Ying},
  title = {Comparing the Accuracy of Two Generated Large Language Models in Identifying Health-Related Rumors or Misconceptions and the Applicability in Health Science Popularization: Proof-of-Concept Study},
  journal = {JMIR Formative Research},
  year = {2024},
  doi = {10.2196/63188}
}
Start for free →