Assessing the Accuracy of Large Language Models on European Guidelines for Cervical Cancer: An In Silico Benchmarking Study
BJOG: An International Journal of Obstetrics & Gynaecology · 2026-03 · 133 (4) · 771-778
DOI: 10.1111/1471-0528.70095 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift BJOG: An International Journal of Obstetrics & Gynaecology, online erschienen am 24.11.2025. Verfasst von Matteo Pavone und 14 weiteren Autor:innen. Beteiligte Einrichtung: UOC Ginecologia Oncologica, Dipartimento di Scienze per la Salute Della Donna e del Bambino e di Sanità Pubblica, Fondazione Policlinico Universitari…. Hauptthemen laut MEDLINE: Uterine Cervical Neoplasms, Practice Guidelines as Topic, Language. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 20 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Objective
Although large language models are increasingly used in clinical and research settings, the validity of the information they provide remains uncertain. This study aimed to evaluate the accuracy, consistency, and reliability of three large language models—ChatGPT 4.0, DeepSeek R1, and Gemini 2.0—in answering cervical cancer‐related questions based on the ESGO/ESTRO/ESP guidelines.
Design
Prospective, comparative in silico benchmarking study.
Setting
Fondazione Policlinico Universitario A. Gemelli, Rome, Italy. Population or Sample: Fifty questions derived from the ESGO/ESTRO/ESP (European Society of Gynaecologic Oncology/European Society for Radiotherapy and Oncology/European Society of Pathology) Guidelines for Cervical Cancer.
Methods
Each question was submitted simultaneously to ChatGPT 4.0, DeepSeek R1, and Gemini 2.0, and re‐entered twice to assess response repeatability. Answers were evaluated for accuracy using a Global Quality Score (GQS) from 1 (poor) to 5 (completely accurate). Consistency (intra‐model response stability) and reliability (alignment with guidelines) were assessed using binary classification. Main Outcome Measures: Median GQS, percentage of GQS 5 responses, consistency between repeated answers, and reliability.
Results
ChatGPT 4.0 achieved the highest performance, with 42% of responses rated GQS 5, followed by Gemini 2.0 (30%) and DeepSeek R1 (28%). DeepSeek R1 and Gemini 2.0 scored lower in median GQS (3.50) compared to ChatGPT 4.0 (4.00). Response consistency varied significantly, with ChatGPT 4.0 and DeepSeek R1 showing differences from Gemini 2.0 ( p = 0.034 and p = 0.044, respectively). No significant difference was observed in reliability ( p = 0.602).
Conclusion
All models demonstrated suboptimal accuracy in aligning with clinical guidelines. ChatGPT 4.0 was the most accurate and consistent whereas DeepSeek R1 underperformed. Despite similar reliability across models, expert oversight remains essential to ensure safe clinical application and prevent misinformation.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- UOC Ginecologia Oncologica, Dipartimento di Scienze per la Salute Della Donna e del Bambino e di Sanità Pubblica, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy.
- Research Institute Against Digestive Cancer, IRCAD Strasbourg, Strasbourg, France.
- Institute of Image-Guided Surgery, IHU Strasbourg, Strasbourg, France.
- ICube, Laboratory of Engineering, Computer Science and Imaging, Department of Robotics, Imaging, Teledetection and Healthcare Technologies, University of Strasbourg, CNRS, UMR 7357, Strasbourg, France.
- Department of Gynecologic Surgery, University Hospitals of Strasbourg, Strasbourg, France.
- Facility of Epidemiology and Biostatistics - Gemelli Generator, Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy.
ORCID-Profile: Matteo Pavone, Nicolò Bizzarri
Zitieren
APA
Pavone, M., Innocenzi, C., Macellari, N., Cantarini, C., Criscione, M., Rosati, A., Lecointre, L., Carcagnì, A., Costantini, B., Marescaux, J., Fagotti, A., Fanfani, F., Cibula, D., Querleu, D., & Bizzarri, N. (2026). Assessing the Accuracy of Large Language Models on European Guidelines for Cervical Cancer: An In Silico Benchmarking Study. BJOG: An International Journal of Obstetrics & Gynaecology, 133(4), 771-778. https://doi.org/10.1111/1471-0528.70095
Vancouver
Pavone M, Innocenzi C, Macellari N, Cantarini C, Criscione M, Rosati A, et al. Assessing the Accuracy of Large Language Models on European Guidelines for Cervical Cancer: An In Silico Benchmarking Study. BJOG: An International Journal of Obstetrics & Gynaecology. 2026;133(4):771-778. doi:10.1111/1471-0528.70095
BibTeX
@article{Pavone2026,
author = {Pavone, Matteo and Innocenzi, Chiara and Macellari, Nicola and Cantarini, Chiara and Criscione, Massimo and Rosati, Andrea and Lecointre, Lise and Carcagnì, Antonella and Costantini, Barbara and Marescaux, Jacques and Fagotti, Anna and Fanfani, Francesco and Cibula, David and Querleu, Denis and Bizzarri, Nicolò},
title = {Assessing the Accuracy of Large Language Models on European Guidelines for Cervical Cancer: An In Silico Benchmarking Study},
journal = {BJOG: An International Journal of Obstetrics & Gynaecology},
year = {2026},
volume = {133},
number = {4},
pages = {771--778},
doi = {10.1111/1471-0528.70095}
}
EUVIMED