EUVIMEDEUVIMED entdecken
Open Access

Detection of Medical Misinformation in Hemangioma Patient Education: Comparative Study of ChatGPT-4o and DeepSeek-R1 Large Language Models

Guoyong Wang, Ye Zhang, Weixin Wang, Yingjie Zhu, Wei Lu, Chaonan Wang, Hui Bi, Xiaonan Yang

JMIR AI · 18.11.2025 · 4 · e76372-e76372

DOI: 10.2196/76372 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR AI, online erschienen am 18.11.2025. Verfasst von Guoyong Wang und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Hemangioma and Vascular Malformation, Plastic Surgery Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, 3…. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

This study examines the capability of large language models (LLMs) in detecting medical rumors, using hemangioma-related information as an example. It compares the performances of ChatGPT-4o and DeepSeek-R1.

Objective

This study aimed to evaluate and compare the accuracy, stability, and expert-rated reliability of 2 LLMs, ChatGPT-4o and DeepSeek-R1, in classifying medical information related to hemangiomas as either “rumors” or “accurate information.” Methods We collected 82 publicly available texts from social media platforms, medical education websites, international guidelines, and journals. Of the 82 items, 47/82 (57%) were labeled as “rumors,” and 35/82 (43%) were labeled as “accurate information.” Three vascular anomaly specialists with extensive clinical experience independently annotated the texts in a double-blinded manner, and disagreements were resolved by arbitration to ensure labeling reliability. Subsequently, these texts were input into ChatGPT-4o and DeepSeek-R1, with each model generating 2 rounds of results under identical instructions. Output stability was assessed using bidirectional encoder representations from transformers–based semantic similarity scores. Classification accuracy, precision, recall, and F 1 -score were calculated to evaluate the performance. Additionally, 2 medical experts independently rated the model outputs using a 5-point scale based on clinical guidelines. Statistical analyses included paired t tests, Wilcoxon signed-rank tests, and bootstrap resampling to compute confidence intervals.

Results

In terms of semantic stability, the similarity distributions for the 2 models largely overlapped, with no statistically significant difference observed (mean difference=−0.003, 95% CI −0.011 to 0.005; P =.30). Regarding classification performance, DeepSeek-R1 achieved higher accuracy (0.963) compared to ChatGPT-4o (0.910), and also performed better in terms of precision (0.978 vs 0.940), recall (0.957 vs 0.894), and F 1 -score (0.967 vs 0.916). Expert evaluations revealed that DeepSeek-R1 significantly outperformed ChatGPT-4o on both “rumor” items (mean difference=0.431; P

Schlagwörter

Stichwörter der Autor:innen

AIartificial intelligenceclassification performancehemangiomalarge language modelsmedical rumorssemantic similarity

Autor:innen und Institutionen

  • Department of Hemangioma and Vascular Malformation, Plastic Surgery Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, 33 Badachu Road, Shijingshan District, Beijing, 100144, China, 86 18810…
  • Department of Internal Medicine, Plastic Surgery Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100144, China.

Zitieren

APA

Wang, G., Zhang, Y., Wang, W., Zhu, Y., Lu, W., Wang, C., Bi, H., & Yang, X. (2025). Detection of Medical Misinformation in Hemangioma Patient Education: Comparative Study of ChatGPT-4o and DeepSeek-R1 Large Language Models. JMIR AI, 4, e76372-e76372. https://doi.org/10.2196/76372

Vancouver

Wang G, Zhang Y, Wang W, Zhu Y, Lu W, Wang C, et al. Detection of Medical Misinformation in Hemangioma Patient Education: Comparative Study of ChatGPT-4o and DeepSeek-R1 Large Language Models. JMIR AI. 2025;4:e76372-e76372. doi:10.2196/76372

BibTeX
@article{Wang2025,
  author = {Wang, Guoyong and Zhang, Ye and Wang, Weixin and Zhu, Yingjie and Lu, Wei and Wang, Chaonan and Bi, Hui and Yang, Xiaonan},
  title = {Detection of Medical Misinformation in Hemangioma Patient Education: Comparative Study of ChatGPT-4o and DeepSeek-R1 Large Language Models},
  journal = {JMIR AI},
  year = {2025},
  volume = {4},
  pages = {e76372--e76372},
  doi = {10.2196/76372}
}
Kostenlos starten →