Comparative performance evaluation of large language models in answering esophageal cancer-related questions: a multi-model assessment study
Frontiers in Digital Health · 07.10.2025 · 7
DOI: 10.3389/fdgth.2025.1670510 ↗Kurz gefasst
Fachartikel in der Zeitschrift Frontiers in Digital Health, online erschienen am 07.10.2025. Verfasst von Zijie He und 12 weiteren Autor:innen. Beteiligte Einrichtung: Department of Thoracic Surgery, Shengli Clinical Medical College of Fujian Medical University, Fujian Provincial Hospital, Fuzhou University Affiliat…. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 28 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
Esophageal cancer has high incidence and mortality rates, leading to increased public demand for accurate information. However, the reliability of online medical information is often questionable. This study systematically compared the accuracy, completeness, and comprehensibility of mainstream large language models (LLMs) in answering esophageal cancer-related questions.
Methods
In total, 65 questions covering fundamental knowledge, preoperative preparation, surgical treatment, and postoperative management were selected. Each model, namely, ChatGPT 5, Claude Sonnet 4.0, DeepSeek-R1, Gemini 2.5 Pro, and Grok-4, was queried independently using standardized prompts. Five senior clinical experts, including three thoracic surgeons, one radiologist, and one medical oncologist, evaluated the responses using a five-point Likert scale. A retesting mechanism was applied for the low-scoring responses, and intraclass correlation coefficients were used to assess the rating consistency. The statistical analyses were conducted using the Friedman test, the Wilcoxon signed-rank test, and the Bonferroni correction.
Results
All the models performed well, with average scores exceeding 4.0. However, the following significant differences emerged: Gemini excelled in accuracy, while ChatGPT led in completeness, particularly in surgical and postoperative contexts. Minor differences appeared in fundamental knowledge, but notable disparities were found in complex areas. Retesting showed improvements in overall quality, yet some responses showed decreased completeness and relevance.
Conclusion
Large language models have considerable potential in answering questions about esophageal cancer, with significant differences in completeness. ChatGPT is more comprehensive in complex scenarios, while Gemini excels in accuracy. This study offers guidance for selecting artificial intelligence tools in clinical settings, advocating for a tiered application strategy tailored to specific scenarios and highlighting the importance of user education to understand the limitations and applicability of LLMs.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Thoracic Surgery, Shengli Clinical Medical College of Fujian Medical University, Fujian Provincial Hospital, Fuzhou University Affiliated Provincial Hospital, Fuzhou, China.
- Department of Cardiothoracic Surgery, Tianjin Medical University General Hospital, Tianjin, China.
- Department of Medical Oncology, Shengli Clinical Medical College of Fujian Medical University, Fujian Provincial Hospital, Fuzhou University Affiliated Provincial Hospital, Fuzhou, China.
- Department of Radiology, Shengli Clinical Medical College of Fujian Medical University, Fujian Provincial Hospital, Fuzhou University Affiliated Provincial Hospital, Fuzhou, China.
Zitieren
APA
He, Z., Zhao, L., Li, G., Wang, J., Cai, S., Tu, P., Chen, J., Wu, J., Zhang, J., Chen, R., Huang, Y., Pan, X., & Chen, W. (2025). Comparative performance evaluation of large language models in answering esophageal cancer-related questions: a multi-model assessment study. Frontiers in Digital Health, 7. https://doi.org/10.3389/fdgth.2025.1670510
Vancouver
He Z, Zhao L, Li G, Wang J, Cai S, Tu P, et al. Comparative performance evaluation of large language models in answering esophageal cancer-related questions: a multi-model assessment study. Frontiers in Digital Health. 2025;7. doi:10.3389/fdgth.2025.1670510
BibTeX
@article{He2025,
author = {He, Zijie and Zhao, Lilan and Li, Genglin and Wang, Jintao and Cai, Songyu and Tu, Pengjie and Chen, Jingbo and Wu, Jianman and Zhang, Juan and Chen, Ruiqi and Huang, Yangyun and Pan, Xiaojie and Chen, Wenshu},
title = {Comparative performance evaluation of large language models in answering esophageal cancer-related questions: a multi-model assessment study},
journal = {Frontiers in Digital Health},
year = {2025},
volume = {7},
doi = {10.3389/fdgth.2025.1670510}
}
EUVIMED