Reasoning or reciting? A temporal contamination audit of large language models in clinical medicine
Journal of the American Medical Informatics Association · 01.09.2026 · 33 (9) · 1617-1626
DOI: 10.1093/jamia/ocag069 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 12.06.2026. Verfasst von Alexander P Sheppert und 3 weiteren Autor:innen. Beteiligte Einrichtung: Internal Medicine, Legacy Health System, Portland, OR 98686, United States. Hauptthemen laut MEDLINE: Large Language Models, Clinical Reasoning, Artificial Intelligence, Clinical Medicine. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 21 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Objective
Evaluate whether large language models reason or simply regurgitate training data in clinical diagnosis. Materials and Methods We audited 2000 clinical case reports from PubMed Central: 1000 from 2021 to 2022 (within training data) and 1000 from 2025 (after training cutoffs). Five frontier LLMs generated diagnoses evaluated by an independent AI judge validated against physician consensus (n = 10 000 evaluations).
Results
Diagnostic accuracy was virtually identical across temporal cohorts (66.8% contaminated vs 66.9% clean), directly contradicting the memorization hypothesis. Lexical similarity was uniformly low (mean ROUGE-L 0.057), and semantic similarity measured by BERTScore showed no memorization signal (F1 0.8182 contaminated vs 0.8195 clean, Δ = +0.0013), confirming that models generate novel reasoning rather than regurgitating training data.
Discussion
This large-scale audit, using both lexical and semantic similarity metrics, provides compelling evidence that LLMs engage in genuine clinical reasoning rather than regurgitating memorized training data.
Conclusion
Models demonstrated equivalent accuracy on cases they could not have seen during training, suggesting they have internalized generalizable medical knowledge rather than memorizing specific cases.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Internal Medicine, Legacy Health System, Portland, OR 98686, United States.
- Internal Medicine, The Vancouver Clinic, Vancouver, WA 98686, United States.
- Artificial Intelligence, Capitol Technology University, Laurel, MD 20708, United States.
- Vascular Surgery, Intermountain Healthcare, Salt Lake City, UT 84604, United States.
- Bioinformatics, University of Utah, Salt Lake City, UT 84112, United States.
Zitieren
APA
Sheppert, A. P., Adams, B., Sheppert, A. D., & Riley, S. (2026). Reasoning or reciting? A temporal contamination audit of large language models in clinical medicine. Journal of the American Medical Informatics Association, 33(9), 1617-1626. https://doi.org/10.1093/jamia/ocag069
Vancouver
Sheppert AP, Adams B, Sheppert AD, Riley S. Reasoning or reciting? A temporal contamination audit of large language models in clinical medicine. Journal of the American Medical Informatics Association. 2026;33(9):1617-1626. doi:10.1093/jamia/ocag069
BibTeX
@article{Sheppert2026,
author = {Sheppert, Alexander P and Adams, Brian and Sheppert, Andrew D and Riley, Sylvia},
title = {Reasoning or reciting? A temporal contamination audit of large language models in clinical medicine},
journal = {Journal of the American Medical Informatics Association},
year = {2026},
volume = {33},
number = {9},
pages = {1617--1626},
doi = {10.1093/jamia/ocag069}
}
EUVIMED