Is one run enough? Reproducibility of flagship large language models across temperature and reasoning settings in biomedical text processing
Journal of the American Medical Informatics Association · 01.06.2026 · 33 (6) · 1179-1184
DOI: 10.1093/jamia/ocag039 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 30.03.2026. Verfasst von Paul Windisch und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiation Oncology, Cantonal Hospital Winterthur, Winterthur, 8401, Switzerland. Hauptthemen laut MEDLINE: Large Language Models, Natural Language Processing. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 13 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Swiss Cancer Research.
Abstract (Originalsprache)
Background
To quantify run-to-run reproducibility of Gemini 3 Flash Preview and GPT-5.2 for trial-success classification across temperature and reasoning/thinking settings and determine whether single-run reporting suffices. Materials and Methods We utilized 250 trial abstracts labeled based on primary endpoint success. We evaluated Gemini across thinking levels (minimal, low, medium, high) and temperatures 0.0-2.0 and GPT-5.2 across reasoning-effort levels (none to x-high) with an additional temperature sweep when reasoning was disabled. Each setting was run 3 times.
Results
Reproducibility was high for Gemini (κ = 0.942-1.000; invalid outputs 0%-1.5%) and GPT-5.2 (κ = 0.984-0.995; no invalid outputs). F1 remained stable (mean/majority vote 0.955-0.971), with marginal gains from majority voting.
Conclusion
For binary biomedical classification with tightly constrained outputs, both models were reproducible across decoding and reasoning settings, suggesting single runs are often sufficient, with minimal replication as a practical stability check.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Radiation Oncology, Cantonal Hospital Winterthur, Winterthur, 8401, Switzerland.
- Department of Radiation Oncology, Inselspital, Bern University Hospital, University of Bern, Bern, 3012, Switzerland.
ORCID-Profile: Paul Windisch
Förderung
- Swiss Cancer Research
Zitieren
APA
Windisch, P., Koechli, C., Dennstädt, F., Aebersold, D. M., Zwahlen, D. R., Förster, R., & Schröder, C. (2026). Is one run enough? Reproducibility of flagship large language models across temperature and reasoning settings in biomedical text processing. Journal of the American Medical Informatics Association, 33(6), 1179-1184. https://doi.org/10.1093/jamia/ocag039
Vancouver
Windisch P, Koechli C, Dennstädt F, Aebersold DM, Zwahlen DR, Förster R, et al. Is one run enough? Reproducibility of flagship large language models across temperature and reasoning settings in biomedical text processing. Journal of the American Medical Informatics Association. 2026;33(6):1179-1184. doi:10.1093/jamia/ocag039
BibTeX
@article{Windisch2026,
author = {Windisch, Paul and Koechli, Carole and Dennstädt, Fabio and Aebersold, Daniel M and Zwahlen, Daniel R and Förster, Robert and Schröder, Christina},
title = {Is one run enough? Reproducibility of flagship large language models across temperature and reasoning settings in biomedical text processing},
journal = {Journal of the American Medical Informatics Association},
year = {2026},
volume = {33},
number = {6},
pages = {1179--1184},
doi = {10.1093/jamia/ocag039}
}
EUVIMED