EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Probabilistic medical predictions of large language models

Bowen Gu, Rishi J Desai, Kueiyu Joshua Lin, Jie Yang

npj Digital Medicine · 19.12.2024 · 7 (1)

DOI: 10.1038/s41746-024-01366-4 ↗

Kurz gefasst

Fachartikel in der Zeitschrift npj Digital Medicine, online erschienen am 19.12.2024. Verfasst von Bowen Gu und 3 weiteren Autor:innen. Beteiligte Einrichtung: Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA. Die Arbeit wurde bisher 57-mal zitiert (Crossref) und nennt 57 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: NLM NIH HHS, NIH HHS, National Institutes of Health.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) have shown promise in clinical applications through prompt engineering, allowing flexible clinical predictions. However, they struggle to produce reliable prediction probabilities, which are crucial for transparency and decision-making. While explicit prompts can lead LLMs to generate probability estimates, their numerical reasoning limitations raise concerns about reliability. We compared explicit probabilities from text generation to implicit probabilities derived from the likelihood of predicting the correct label token. Across six advanced open-source LLMs and five medical datasets, explicit probabilities consistently underperformed implicit probabilities in discrimination, precision, and recall. This discrepancy is more pronounced with smaller LLMs and imbalanced datasets, highlighting the need for cautious interpretation, improved probability estimation methods, and further research for clinical use of LLMs.

Autor:innen und Institutionen

  • Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
  • Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA. jyang66@bwh.harvard.edu.
  • Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA. jyang66@bwh.harvard.edu.
  • Broad Institute of MIT and Harvard, Cambridge, MA, USA. jyang66@bwh.harvard.edu.
  • Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Cambridge, MA, USA. jyang66@bwh.harvard.edu.

Förderung

  • NLM NIH HHS
  • NIH HHS
  • National Institutes of Health

Zitieren

APA

Gu, B., Desai, R. J., Lin, K. J., & Yang, J. (2024). Probabilistic medical predictions of large language models. npj Digital Medicine, 7(1). https://doi.org/10.1038/s41746-024-01366-4

Vancouver

Gu B, Desai RJ, Lin KJ, Yang J. Probabilistic medical predictions of large language models. npj Digital Medicine. 2024;7(1). doi:10.1038/s41746-024-01366-4

BibTeX
@article{Gu2024,
  author = {Gu, Bowen and Desai, Rishi J and Lin, Kueiyu Joshua and Yang, Jie},
  title = {Probabilistic medical predictions of large language models},
  journal = {npj Digital Medicine},
  year = {2024},
  volume = {7},
  number = {1},
  doi = {10.1038/s41746-024-01366-4}
}
Kostenlos starten →