EUVIMEDEUVIMED entdecken
VergleichsstudieLeitlinieOpen Access · CC BY 4.0

Assessing the Accuracy of Large Language Models on European Guidelines for Cervical Cancer: An In Silico Benchmarking Study

Matteo Pavone iD, Chiara Innocenzi, Nicola Macellari, Chiara Cantarini, Massimo Criscione, Andrea Rosati, Lise Lecointre, Antonella Carcagnì, Barbara Costantini, Jacques Marescaux, Anna Fagotti, Francesco Fanfani und 3 weitere

BJOG: An International Journal of Obstetrics & Gynaecology · 2026-03 · 133 (4) · 771-778

DOI: 10.1111/1471-0528.70095 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift BJOG: An International Journal of Obstetrics & Gynaecology, online erschienen am 24.11.2025. Verfasst von Matteo Pavone und 14 weiteren Autor:innen. Beteiligte Einrichtung: UOC Ginecologia Oncologica, Dipartimento di Scienze per la Salute Della Donna e del Bambino e di Sanità Pubblica, Fondazione Policlinico Universitari…. Hauptthemen laut MEDLINE: Uterine Cervical Neoplasms, Practice Guidelines as Topic, Language. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 20 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Although large language models are increasingly used in clinical and research settings, the validity of the information they provide remains uncertain. This study aimed to evaluate the accuracy, consistency, and reliability of three large language models—ChatGPT 4.0, DeepSeek R1, and Gemini 2.0—in answering cervical cancer‐related questions based on the ESGO/ESTRO/ESP guidelines.

Design

Prospective, comparative in silico benchmarking study.

Setting

Fondazione Policlinico Universitario A. Gemelli, Rome, Italy. Population or Sample: Fifty questions derived from the ESGO/ESTRO/ESP (European Society of Gynaecologic Oncology/European Society for Radiotherapy and Oncology/European Society of Pathology) Guidelines for Cervical Cancer.

Methods

Each question was submitted simultaneously to ChatGPT 4.0, DeepSeek R1, and Gemini 2.0, and re‐entered twice to assess response repeatability. Answers were evaluated for accuracy using a Global Quality Score (GQS) from 1 (poor) to 5 (completely accurate). Consistency (intra‐model response stability) and reliability (alignment with guidelines) were assessed using binary classification. Main Outcome Measures: Median GQS, percentage of GQS 5 responses, consistency between repeated answers, and reliability.

Results

ChatGPT 4.0 achieved the highest performance, with 42% of responses rated GQS 5, followed by Gemini 2.0 (30%) and DeepSeek R1 (28%). DeepSeek R1 and Gemini 2.0 scored lower in median GQS (3.50) compared to ChatGPT 4.0 (4.00). Response consistency varied significantly, with ChatGPT 4.0 and DeepSeek R1 showing differences from Gemini 2.0 ( p = 0.034 and p = 0.044, respectively). No significant difference was observed in reliability ( p = 0.602).

Conclusion

All models demonstrated suboptimal accuracy in aligning with clinical guidelines. ChatGPT 4.0 was the most accurate and consistent whereas DeepSeek R1 underperformed. Despite similar reliability across models, expert oversight remains essential to ensure safe clinical application and prevent misinformation.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Uterine Cervical NeoplasmsPractice Guidelines as TopicLanguageHumansFemaleBenchmarkingReproducibility of ResultsProspective StudiesComputer SimulationEuropeSurveys and QuestionnairesLarge Language Models

Stichwörter der Autor:innen

artificial intelligencecervical cancerchatgptdeepseekdigital surgerygeminilarge language models

Autor:innen und Institutionen

  • UOC Ginecologia Oncologica, Dipartimento di Scienze per la Salute Della Donna e del Bambino e di Sanità Pubblica, Fondazione Policlinico Universitario A. Gemelli, IRCCS, Rome, Italy.
  • Research Institute Against Digestive Cancer, IRCAD Strasbourg, Strasbourg, France.
  • Institute of Image-Guided Surgery, IHU Strasbourg, Strasbourg, France.
  • ICube, Laboratory of Engineering, Computer Science and Imaging, Department of Robotics, Imaging, Teledetection and Healthcare Technologies, University of Strasbourg, CNRS, UMR 7357, Strasbourg, France.
  • Department of Gynecologic Surgery, University Hospitals of Strasbourg, Strasbourg, France.
  • Facility of Epidemiology and Biostatistics - Gemelli Generator, Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy.

ORCID-Profile: Matteo Pavone, Nicolò Bizzarri

Zitieren

APA

Pavone, M., Innocenzi, C., Macellari, N., Cantarini, C., Criscione, M., Rosati, A., Lecointre, L., Carcagnì, A., Costantini, B., Marescaux, J., Fagotti, A., Fanfani, F., Cibula, D., Querleu, D., & Bizzarri, N. (2026). Assessing the Accuracy of Large Language Models on European Guidelines for Cervical Cancer: An In Silico Benchmarking Study. BJOG: An International Journal of Obstetrics & Gynaecology, 133(4), 771-778. https://doi.org/10.1111/1471-0528.70095

Vancouver

Pavone M, Innocenzi C, Macellari N, Cantarini C, Criscione M, Rosati A, et al. Assessing the Accuracy of Large Language Models on European Guidelines for Cervical Cancer: An In Silico Benchmarking Study. BJOG: An International Journal of Obstetrics & Gynaecology. 2026;133(4):771-778. doi:10.1111/1471-0528.70095

BibTeX
@article{Pavone2026,
  author = {Pavone, Matteo and Innocenzi, Chiara and Macellari, Nicola and Cantarini, Chiara and Criscione, Massimo and Rosati, Andrea and Lecointre, Lise and Carcagnì, Antonella and Costantini, Barbara and Marescaux, Jacques and Fagotti, Anna and Fanfani, Francesco and Cibula, David and Querleu, Denis and Bizzarri, Nicolò},
  title = {Assessing the Accuracy of Large Language Models on European Guidelines for Cervical Cancer: An In Silico Benchmarking Study},
  journal = {BJOG: An International Journal of Obstetrics & Gynaecology},
  year = {2026},
  volume = {133},
  number = {4},
  pages = {771--778},
  doi = {10.1111/1471-0528.70095}
}
Kostenlos starten →