Establishing best practices in large language model research: an application to repeat prompting
Journal of the American Medical Informatics Association · 01.02.2025 · 32 (2) · 386-390
DOI: 10.1093/jamia/ocae294 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 04.12.2024. Verfasst von Robert J Gallo und 3 weiteren Autor:innen. Beteiligte Einrichtung: Center for Innovation to Implementation, VA Palo Alto Health Care System, Menlo Park, CA 94025, United States. Hauptthemen laut MEDLINE: Abstracting and Indexing, Biomedical Research. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 20 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Institute of Allergy and Infectious Diseases, NIDA NIH HHS, Gordon and Betty Moore Foundation.
Abstract (Originalsprache)
Objectives
We aimed to demonstrate the importance of establishing best practices in large language model research, using repeat prompting as an illustrative example. Materials and Methods Using data from a prior study investigating potential model bias in peer review of medical abstracts, we compared methods that ignore correlation in model outputs from repeated prompting with a random effects method that accounts for this correlation.
Results
High correlation within groups was found when repeatedly prompting the model, with intraclass correlation coefficient of 0.69. Ignoring the inherent correlation in the data led to over 100-fold inflation of effective sample size. After appropriately accounting for this issue, the authors’ results reverse from a small but highly significant finding to no evidence of model bias.
Discussion
The establishment of best practices for LLM research is urgently needed, as demonstrated in this case where accounting for repeat prompting in analyses was critical for accurate study conclusions.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Center for Innovation to Implementation, VA Palo Alto Health Care System, Menlo Park, CA 94025, United States.
- Department of Health Policy, Stanford University, Stanford, CA 94305, United States.
- Department of Epidemiology and Population Health, Stanford University, Stanford, CA 94305, United States.
- Division of Hospital Medicine, Stanford University, Stanford, CA 94305, United States.
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA 94304, United States.
- Clinical Excellence Research Center, Stanford University, Stanford, CA 94305, United States.
ORCID-Profile: Robert J Gallo, Michael Baiocchi
Förderung
- National Institute of Allergy and Infectious Diseases
- NIDA NIH HHS
- Gordon and Betty Moore Foundation
- American Heart Association-Strategically Focused Research Network-Diversity in Clinical Trials
- United States Government
- Medical Informatics
- Stanford Artificial Intelligence in Medicine and Imaging-Human-Centered Artificial Intelligence
- National Institute on Drug Abuse Clinical Trials Network
Zitieren
APA
Gallo, R. J., Baiocchi, M., Savage, T. R., & Chen, J. H. (2025). Establishing best practices in large language model research: an application to repeat prompting. Journal of the American Medical Informatics Association, 32(2), 386-390. https://doi.org/10.1093/jamia/ocae294
Vancouver
Gallo RJ, Baiocchi M, Savage TR, Chen JH. Establishing best practices in large language model research: an application to repeat prompting. Journal of the American Medical Informatics Association. 2025;32(2):386-390. doi:10.1093/jamia/ocae294
BibTeX
@article{Gallo2025,
author = {Gallo, Robert J and Baiocchi, Michael and Savage, Thomas R and Chen, Jonathan H},
title = {Establishing best practices in large language model research: an application to repeat prompting},
journal = {Journal of the American Medical Informatics Association},
year = {2025},
volume = {32},
number = {2},
pages = {386--390},
doi = {10.1093/jamia/ocae294}
}
EUVIMED