EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Comparing the Accuracy of Two Generated Large Language Models in Identifying Health-Related Rumors or Misconceptions and the Applicability in Health Science Popularization: Proof-of-Concept Study

Yuan Luo iD, Yiqun Miao iD, Yuhan Zhao iD, Jiawei Li iD, Yuling Chen iD, Yuexue Yue iD, Ying Wu iD

JMIR Formative Research · 2024

DOI: 10.2196/63188 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift JMIR Formative Research, 2024. Verfasst von Yuan Luo und 6 weiteren Autor:innen. Beteiligte Einrichtung: School of Nursing, Capital Medical University, 10 Xitoutiao, Youanmen Wai, Fengtai District, Beijing, 100069, China, 86 10839117. Hauptthemen laut MEDLINE: Social Media. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Health-related rumors and misconceptions are spreading at an alarming rate, fueled by the rapid development of the internet and the exponential growth of social media platforms. This phenomenon has become a pressing global concern, as the dissemination of false information can have severe consequences, including widespread panic, social instability, and even public health crises.

Objective

The aim of the study is to compare the accuracy of rumor identification and the effectiveness of health science popularization between 2 generated large language models in Chinese (GPT-4 by OpenAI and Enhanced Representation through Knowledge Integration Bot [ERNIE Bot] 4.0 by Baidu).

Methods

In total, 20 health rumors and misconceptions, along with 10 health truths, were randomly inputted into GPT-4 and ERNIE Bot 4.0. We prompted them to determine whether the statements were rumors or misconceptions and provide explanations for their judgment. Further, we asked them to generate a health science popularization essay. We evaluated the outcomes in terms of accuracy, effectiveness, readability, and applicability. Accuracy was assessed by the rate of correctly identifying health-related rumors, misconceptions, and truths. Effectiveness was determined by the accuracy of the generated explanation, which was assessed collaboratively by 2 research team members with a PhD in nursing. Readability was calculated by the readability formula of Chinese health education materials. Applicability was evaluated by the Chinese Suitability Assessment of Materials.

Results

GPT-4 and ERNIE Bot 4.0 correctly identified all health rumors and misconceptions (100% accuracy rate). For truths, the accuracy rate was 70% (7/10) and 100% (10/10), respectively. Both mostly provided widely recognized viewpoints without obvious errors. The average readability score for the health essays was 2.92 (SD 0.85) for GPT-4 and 3.02 (SD 0.84) for ERNIE Bot 4.0 ( P =.65). For applicability, except for the content and cultural appropriateness category, significant differences were observed in the total score and scores in other dimensions between them ( P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Social MediaHumansProof of Concept StudyChinaLanguage

Stichwörter der Autor:innen

LLMaccuracyapplicabilityeducationeffectivenesshealth educationhealth relatedhealth sciencehealth science popularizationlarge language modelmisconceptionproof of conceptrumor

Autor:innen und Institutionen

  • School of Nursing, Capital Medical University, 10 Xitoutiao, Youanmen Wai, Fengtai District, Beijing, 100069, China, 86 10839117.
  • School of Nursing, Johns Hopkins University, Baltimore, MD, United States.

ORCID-Profile: Yuan Luo, Yiqun Miao, Yuhan Zhao, Jiawei Li, Yuling Chen, Yuexue Yue, Ying Wu

Zitieren

APA

Luo, Y., Miao, Y., Zhao, Y., Li, J., Chen, Y., Yue, Y., & Wu, Y. (2024). Comparing the Accuracy of Two Generated Large Language Models in Identifying Health-Related Rumors or Misconceptions and the Applicability in Health Science Popularization: Proof-of-Concept Study. JMIR Formative Research. https://doi.org/10.2196/63188

Vancouver

Luo Y, Miao Y, Zhao Y, Li J, Chen Y, Yue Y, et al. Comparing the Accuracy of Two Generated Large Language Models in Identifying Health-Related Rumors or Misconceptions and the Applicability in Health Science Popularization: Proof-of-Concept Study. JMIR Formative Research. 2024. doi:10.2196/63188

BibTeX
@article{Luo2024,
  author = {Luo, Yuan and Miao, Yiqun and Zhao, Yuhan and Li, Jiawei and Chen, Yuling and Yue, Yuexue and Wu, Ying},
  title = {Comparing the Accuracy of Two Generated Large Language Models in Identifying Health-Related Rumors or Misconceptions and the Applicability in Health Science Popularization: Proof-of-Concept Study},
  journal = {JMIR Formative Research},
  year = {2024},
  doi = {10.2196/63188}
}
Kostenlos starten →