Toward expert-level medical question answering with large language models
Nature Medicine · 2025-03 · 31 (3) · 943-950
DOI: 10.1038/s41591-024-03423-7 ↗Kurz gefasst
Fachartikel in der Zeitschrift Nature Medicine, online erschienen am 08.01.2025. Verfasst von Karan Singhal und 34 weiteren Autor:innen. Beteiligte Einrichtung: Google Research, Mountain View, CA, USA. Die Arbeit wurde bisher 1161-mal zitiert (Crossref) und nennt 63 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: NCATS NIH HHS.
Abstract (Originalsprache)
Large language models (LLMs) have shown promise in medical question answering, with Med-PaLM being the first to exceed a 'passing' score in United States Medical Licensing Examination style questions. However, challenges remain in long-form medical question answering and handling real-world workflows. Here, we present Med-PaLM 2, which bridges these gaps with a combination of base LLM improvements, medical domain fine-tuning and new strategies for improving reasoning and grounding through ensemble refinement and chain of retrieval. Med-PaLM 2 scores up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19%, and demonstrates dramatic performance increases across MedMCQA, PubMedQA and MMLU clinical topics datasets. Our detailed human evaluations framework shows that physicians prefer Med-PaLM 2 answers to those from other physicians on eight of nine clinical axes. Med-PaLM 2 also demonstrates significant improvements over its predecessor across all evaluation metrics, particularly on new adversarial datasets designed to probe LLM limitations (P < 0.001). In a pilot study using real-world medical questions, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time. While specialist answers were still preferred overall, both specialists and generalists rated Med-PaLM 2 to be as safe as physician answers, demonstrating its growing potential in real-world medical applications.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Google Research, Mountain View, CA, USA.
- Google DeepMind, Mountain View, CA, USA.
- Department of Emergency Medicine, Stanford University School of Medicine, Stanford, CA, USA.
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA.
- Division of Hospital Medicine, Stanford University, Stanford, CA, USA.
- Clinical Excellence Research Center, Stanford University, Stanford, CA, USA.
Förderung
- NCATS NIH HHS
Zitieren
APA
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., Neal, D., Rashid, Q. M., Schaekermann, M., Wang, A., Dash, D., Chen, J. H., Shah, N. H., Lachgar, S., Mansfield, P. A., …, & Natarajan, V. (2025). Toward expert-level medical question answering with large language models. Nature Medicine, 31(3), 943-950. https://doi.org/10.1038/s41591-024-03423-7
Vancouver
Singhal K, Tu T, Gottweis J, Sayres R, Wulczyn E, Amin M, et al. Toward expert-level medical question answering with large language models. Nature Medicine. 2025;31(3):943-950. doi:10.1038/s41591-024-03423-7
BibTeX
@article{Singhal2025,
author = {Singhal, Karan and Tu, Tao and Gottweis, Juraj and Sayres, Rory and Wulczyn, Ellery and Amin, Mohamed and Hou, Le and Clark, Kevin and Pfohl, Stephen R and Cole-Lewis, Heather and Neal, Darlene and Rashid, Qazi Mamunur and Schaekermann, Mike and Wang, Amy and Dash, Dev and Chen, Jonathan H and Shah, Nigam H and Lachgar, Sami and Mansfield, Philip Andrew and Prakash, Sushant and Green, Bradley and Dominowska, Ewa and Agüera Y Arcas, Blaise and Tomašev, Nenad and Liu, Yun and Wong, Renee and Semturs, Christopher and Mahdavi, S Sara and Barral, Joelle K and Webster, Dale R and Corrado, Greg S and Matias, Yossi and Azizi, Shekoofeh and Karthikesalingam, Alan and Natarajan, Vivek},
title = {Toward expert-level medical question answering with large language models},
journal = {Nature Medicine},
year = {2025},
volume = {31},
number = {3},
pages = {943--950},
doi = {10.1038/s41591-024-03423-7}
}
EUVIMED