EUVIMEDEUVIMED entdecken
LeitlinieOpen Access

Evaluating the Accuracy of ChatGPT‐4o in Addressing Complex Clinical Questions Based on NCCN Guidelines for Rectal Adenocarcinoma

Ryan Meyer iD, Tamir E Bresler iD, Kevin M Palmer iD, Tyler Wilson iD, Shivam Pandya, Manabu Fujita

Journal of Surgical Oncology · 2026

DOI: 10.1002/jso.70314 ↗

Kurz gefasst

Leitlinie in der Zeitschrift Journal of Surgical Oncology, 2026. Verfasst von Ryan Meyer und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Surgery, Los Robles Regional Medical Center, Thousand Oaks, California, USA. Hauptthemen laut MEDLINE: Rectal Neoplasms, Adenocarcinoma, Practice Guidelines as Topic, Guideline Adherence. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 15 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

The management of rectal adenocarcinoma requires navigation of complex, branching guideline pathways encompassing neoadjuvant sequencing, surgical approach, organ preservation, and surveillance, yet real‐world guideline adherence remains as low as 60–70%. The ability of current‐generation large language models (LLMs) to accurately navigate these decision points has not been fully characterized.

Methods

In this cross‐sectional, vignette‐based study, 135 clinical questions were constructed from 45 pages of NCCN Rectal Cancer Guidelines (Version 4.2024). ChatGPT‐4o was queried using standardized prompts with up to 3 clarifying questions permitted per query. Responses were independently evaluated by two physician raters on a 5‐point Likert scale, with potential discrepancies adjudicated by a board‐certified surgical oncologist. Primary outcomes were the proportion of responses rated Correct (score ≥ 3) and Accurate (score ≥ 4). Inter‐rater reliability was assessed using Cohen's kappa, and subgroup analysis was performed across clinical domains using the Kruskal‐Wallis test.

Results

Of 135 questions, 127 (94.1%; 95% CI, 88.7–97.0%) were Correct and 121 (89.6%; 95% CI, 83.3–93.7%) were Accurate. One hundred two responses (75.6%) were completely correct without additional prompting. Performance was consistent across clinical domains (Kruskal‐Wallis H = 0.530, p = 0.767). Inter‐rater agreement was perfect (κ = 1.0). Eight responses (5.9%) contained partially or wholly incorrect information, with errors concentrated in multi‐step conditional treatment decision points.

Conclusion

ChatGPT‐4o demonstrates high concordance with NCCN rectal cancer guidelines across all evaluated clinical domains with notable improvement over prior ChatGPT iterations evaluated by our group. The concentration of errors in complex conditional treatment algorithms suggests that LLMs excel at discrete factual recall but may struggle with multi‐step reasoning under clinical uncertainty. Prospective validation using real‐world clinical data and comparison with multidisciplinary tumor board recommendations remain necessary prior to clinical integration.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Rectal NeoplasmsAdenocarcinomaPractice Guidelines as TopicGuideline AdherenceHumansLarge Language ModelsCross-Sectional StudiesGenerative Artificial IntelligenceReproducibility of Results

Stichwörter der Autor:innen

NCCN guidelinesartificial intelligenceclinical decision supportlarge language modelsrectal cancer

Autor:innen und Institutionen

  • Department of Surgery, Los Robles Regional Medical Center, Thousand Oaks, California, USA.

ORCID-Profile: Ryan Meyer, Tamir E Bresler, Kevin M Palmer, Tyler Wilson

Zitieren

APA

Meyer, R., Bresler, T. E., Palmer, K. M., Wilson, T., Pandya, S., & Fujita, M. (2026). Evaluating the Accuracy of ChatGPT‐4o in Addressing Complex Clinical Questions Based on NCCN Guidelines for Rectal Adenocarcinoma. Journal of Surgical Oncology. https://doi.org/10.1002/jso.70314

Vancouver

Meyer R, Bresler TE, Palmer KM, Wilson T, Pandya S, Fujita M. Evaluating the Accuracy of ChatGPT‐4o in Addressing Complex Clinical Questions Based on NCCN Guidelines for Rectal Adenocarcinoma. Journal of Surgical Oncology. 2026. doi:10.1002/jso.70314

BibTeX
@article{Meyer2026,
  author = {Meyer, Ryan and Bresler, Tamir E and Palmer, Kevin M and Wilson, Tyler and Pandya, Shivam and Fujita, Manabu},
  title = {Evaluating the Accuracy of ChatGPT‐4o in Addressing Complex Clinical Questions Based on NCCN Guidelines for Rectal Adenocarcinoma},
  journal = {Journal of Surgical Oncology},
  year = {2026},
  doi = {10.1002/jso.70314}
}
Kostenlos starten →