EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Is one run enough? Reproducibility of flagship large language models across temperature and reasoning settings in biomedical text processing

Paul Windisch iD, Carole Koechli, Fabio Dennstädt, Daniel M Aebersold, Daniel R Zwahlen, Robert Förster, Christina Schröder

Journal of the American Medical Informatics Association · 01.06.2026 · 33 (6) · 1179-1184

DOI: 10.1093/jamia/ocag039 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 30.03.2026. Verfasst von Paul Windisch und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiation Oncology, Cantonal Hospital Winterthur, Winterthur, 8401, Switzerland. Hauptthemen laut MEDLINE: Large Language Models, Natural Language Processing. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 13 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Swiss Cancer Research.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

To quantify run-to-run reproducibility of Gemini 3 Flash Preview and GPT-5.2 for trial-success classification across temperature and reasoning/thinking settings and determine whether single-run reporting suffices. Materials and Methods We utilized 250 trial abstracts labeled based on primary endpoint success. We evaluated Gemini across thinking levels (minimal, low, medium, high) and temperatures 0.0-2.0 and GPT-5.2 across reasoning-effort levels (none to x-high) with an additional temperature sweep when reasoning was disabled. Each setting was run 3 times.

Results

Reproducibility was high for Gemini (κ = 0.942-1.000; invalid outputs 0%-1.5%) and GPT-5.2 (κ = 0.984-0.995; no invalid outputs). F1 remained stable (mean/majority vote 0.955-0.971), with marginal gains from majority voting.

Conclusion

For binary biomedical classification with tightly constrained outputs, both models were reproducible across decoding and reasoning settings, suggesting single runs are often sufficient, with minimal replication as a practical stability check.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsNatural Language ProcessingReproducibility of ResultsHumansClinical Trials as Topic

Stichwörter der Autor:innen

large language modelsnatural language processingreasoning modelsreproducibilitytemperature

Autor:innen und Institutionen

  • Department of Radiation Oncology, Cantonal Hospital Winterthur, Winterthur, 8401, Switzerland.
  • Department of Radiation Oncology, Inselspital, Bern University Hospital, University of Bern, Bern, 3012, Switzerland.

ORCID-Profile: Paul Windisch

Förderung

  • Swiss Cancer Research

Zitieren

APA

Windisch, P., Koechli, C., Dennstädt, F., Aebersold, D. M., Zwahlen, D. R., Förster, R., & Schröder, C. (2026). Is one run enough? Reproducibility of flagship large language models across temperature and reasoning settings in biomedical text processing. Journal of the American Medical Informatics Association, 33(6), 1179-1184. https://doi.org/10.1093/jamia/ocag039

Vancouver

Windisch P, Koechli C, Dennstädt F, Aebersold DM, Zwahlen DR, Förster R, et al. Is one run enough? Reproducibility of flagship large language models across temperature and reasoning settings in biomedical text processing. Journal of the American Medical Informatics Association. 2026;33(6):1179-1184. doi:10.1093/jamia/ocag039

BibTeX
@article{Windisch2026,
  author = {Windisch, Paul and Koechli, Carole and Dennstädt, Fabio and Aebersold, Daniel M and Zwahlen, Daniel R and Förster, Robert and Schröder, Christina},
  title = {Is one run enough? Reproducibility of flagship large language models across temperature and reasoning settings in biomedical text processing},
  journal = {Journal of the American Medical Informatics Association},
  year = {2026},
  volume = {33},
  number = {6},
  pages = {1179--1184},
  doi = {10.1093/jamia/ocag039}
}
Kostenlos starten →