EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Establishing best practices in large language model research: an application to repeat prompting

Robert J Gallo iD, Michael Baiocchi iD, Thomas R Savage, Jonathan H Chen

Journal of the American Medical Informatics Association · 01.02.2025 · 32 (2) · 386-390

DOI: 10.1093/jamia/ocae294 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 04.12.2024. Verfasst von Robert J Gallo und 3 weiteren Autor:innen. Beteiligte Einrichtung: Center for Innovation to Implementation, VA Palo Alto Health Care System, Menlo Park, CA 94025, United States. Hauptthemen laut MEDLINE: Abstracting and Indexing, Biomedical Research. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 20 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Institute of Allergy and Infectious Diseases, NIDA NIH HHS, Gordon and Betty Moore Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

We aimed to demonstrate the importance of establishing best practices in large language model research, using repeat prompting as an illustrative example. Materials and Methods Using data from a prior study investigating potential model bias in peer review of medical abstracts, we compared methods that ignore correlation in model outputs from repeated prompting with a random effects method that accounts for this correlation.

Results

High correlation within groups was found when repeatedly prompting the model, with intraclass correlation coefficient of 0.69. Ignoring the inherent correlation in the data led to over 100-fold inflation of effective sample size. After appropriately accounting for this issue, the authors’ results reverse from a small but highly significant finding to no evidence of model bias.

Discussion

The establishment of best practices for LLM research is urgently needed, as demonstrated in this case where accounting for repeat prompting in analyses was critical for accurate study conclusions.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Abstracting and IndexingBiomedical ResearchHumansLarge Language Models

Stichwörter der Autor:innen

large language modelmultilevel analysispeer review

Autor:innen und Institutionen

  • Center for Innovation to Implementation, VA Palo Alto Health Care System, Menlo Park, CA 94025, United States.
  • Department of Health Policy, Stanford University, Stanford, CA 94305, United States.
  • Department of Epidemiology and Population Health, Stanford University, Stanford, CA 94305, United States.
  • Division of Hospital Medicine, Stanford University, Stanford, CA 94305, United States.
  • Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA 94304, United States.
  • Clinical Excellence Research Center, Stanford University, Stanford, CA 94305, United States.

ORCID-Profile: Robert J Gallo, Michael Baiocchi

Förderung

  • National Institute of Allergy and Infectious Diseases
  • NIDA NIH HHS
  • Gordon and Betty Moore Foundation
  • American Heart Association-Strategically Focused Research Network-Diversity in Clinical Trials
  • United States Government
  • Medical Informatics
  • Stanford Artificial Intelligence in Medicine and Imaging-Human-Centered Artificial Intelligence
  • National Institute on Drug Abuse Clinical Trials Network

Zitieren

APA

Gallo, R. J., Baiocchi, M., Savage, T. R., & Chen, J. H. (2025). Establishing best practices in large language model research: an application to repeat prompting. Journal of the American Medical Informatics Association, 32(2), 386-390. https://doi.org/10.1093/jamia/ocae294

Vancouver

Gallo RJ, Baiocchi M, Savage TR, Chen JH. Establishing best practices in large language model research: an application to repeat prompting. Journal of the American Medical Informatics Association. 2025;32(2):386-390. doi:10.1093/jamia/ocae294

BibTeX
@article{Gallo2025,
  author = {Gallo, Robert J and Baiocchi, Michael and Savage, Thomas R and Chen, Jonathan H},
  title = {Establishing best practices in large language model research: an application to repeat prompting},
  journal = {Journal of the American Medical Informatics Association},
  year = {2025},
  volume = {32},
  number = {2},
  pages = {386--390},
  doi = {10.1093/jamia/ocae294}
}
Kostenlos starten →