EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A statistical framework for evaluating the repeatability and reproducibility of large language models

Cathy Shyr iD, Boyu Ren iD, Chih-Yuan Hsu iD, Minh-Ha Nguyen, Mike Enger, Chao Yan iD, Rory J Tinker iD, Thomas A Cassini, Rizwan Hamid, Adam Wright iD, Lisa Bastarache, Josh F Peterson und 3 weitere

Journal of the American Medical Informatics Association · 05.11.2026

DOI: 10.1093/jamia/ocag186 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 05.11.2026. Verfasst von Cathy Shyr und 14 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States. Die Arbeit nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Institutes of Health Common Fund, NHGRI NIH HHS, NINDS NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Systematic evaluation of variability in large language model (LLM)-generated outputs is critical for assessing their reliability. We developed a general statistical framework and Python package (llmReproScore) for quantifying the repeatability and reproducibility of LLM outputs. Materials and Methods Following US Food and Drug Administration terminology, our framework establishes four metrics for quantifying LLM repeatability and reproducibility along two dimensions: semantic and internal. Semantic metrics quantify variability in the meaning of LLM-generated outputs, whereas internal metrics quantify variability in token-generation certainty. We validated these metrics, and as an illustrative use case, applied them to diagnostic reasoning outputs generated across LLMs and prompting strategies using US Medical Licensing Examination questions and 90 real-world rare disease cases from the Undiagnosed Diseases Network.

Results

Validation results showed that semantic scores aligned with human ratings (Spearman ρ = 0.88 for repeatability and 0.66 for reproducibility) and confirmed that internal metrics capture variability in token-generation certainty. In the illustrative use case, LLM repeatability was generally not associated with accuracy, suggesting that LLM output variability and accuracy capture distinct dimensions of model performance.

Discussion

The proposed metrics can be used to characterize LLM output variability and systematically compare models, prompts, and generation settings. These comparisons can inform more rigorous selection of LLM configurations based on repeatability and reproducibility.

Conclusion

We developed and validated generalizable metrics for quantifying LLM output variability. The proposed metrics can complement accuracy-based evaluation to inform experimental design and support more rigorous comparison and selection of models, prompts, and generation settings.

Schlagwörter

Stichwörter der Autor:innen

artificial intelligencelarge language modelrepeatabilityreproducibilityvariability

Autor:innen und Institutionen

  • Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States.
  • Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN 37203, United States.
  • Department of Pediatrics, Vanderbilt University Medical Center, Nashville, TN 37203, United States.
  • Laboratory for Psychiatric Biostatistics, McLean Hospital, Belmont, MA 02478, United States.
  • Department of Epidemiology, Vanderbilt University, Nashville, TN 37203, United States.
  • Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States.

ORCID-Profile: Cathy Shyr, Boyu Ren, Chih-Yuan Hsu, Chao Yan, Rory J Tinker, Adam Wright, Hua Xu

Förderung

  • National Institutes of Health Common Fund
  • NHGRI NIH HHS
  • NINDS NIH HHS
  • National Human Genome Research Institute
  • National Institute of Neurological Disorders and Stroke

Zitieren

APA

Shyr, C., Ren, B., Hsu, C. Y., Nguyen, M. H., Enger, M., Yan, C., Tinker, R. J., Cassini, T. A., Hamid, R., Wright, A., Bastarache, L., Peterson, J. F., Malin, B. A., Xu, H., & Undiagnosed Diseases Network (2026). A statistical framework for evaluating the repeatability and reproducibility of large language models. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag186

Vancouver

Shyr C, Ren B, Hsu CY, Nguyen MH, Enger M, Yan C, et al. A statistical framework for evaluating the repeatability and reproducibility of large language models. Journal of the American Medical Informatics Association. 2026. doi:10.1093/jamia/ocag186

BibTeX
@article{Shyr2026,
  author = {Shyr, Cathy and Ren, Boyu and Hsu, Chih-Yuan and Nguyen, Minh-Ha and Enger, Mike and Yan, Chao and Tinker, Rory J and Cassini, Thomas A and Hamid, Rizwan and Wright, Adam and Bastarache, Lisa and Peterson, Josh F and Malin, Bradley A and Xu, Hua and Undiagnosed Diseases Network, Undiagnosed Diseases Network},
  title = {A statistical framework for evaluating the repeatability and reproducibility of large language models},
  journal = {Journal of the American Medical Informatics Association},
  year = {2026},
  doi = {10.1093/jamia/ocag186}
}
Kostenlos starten →