Reasoning with large language models for medical question answering
Journal of the American Medical Informatics Association · 01.09.2024 · 31 (9) · 1964-1975
DOI: 10.1093/jamia/ocae131 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 03.07.2024. Verfasst von Mary M Lucas und 3 weiteren Autor:innen. Beteiligte Einrichtung: College of Computing and Informatics, Drexel University, Philadelphia, PA 19104, United States. Hauptthemen laut MEDLINE: Natural Language Processing. Die Arbeit wurde bisher 76-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: National Science Foundation, Department of Defense.
Abstract (Originalsprache)
Objectives
To investigate approaches of reasoning with large language models (LLMs) and to propose a new prompting approach, ensemble reasoning, to improve medical question answering performance with refined reasoning and reduced inconsistency. Materials and Methods We used multiple choice questions from the USMLE Sample Exam question files on 2 closed-source commercial and 1 open-source clinical LLM to evaluate our proposed approach ensemble reasoning.
Results
On GPT-3.5 turbo and Med42-70B, our proposed ensemble reasoning approach outperformed zero-shot chain-of-thought with self-consistency on Steps 1, 2, and 3 questions (+3.44%, +4.00%, and +2.54%) and (2.3%, 5.00%, and 4.15%), respectively. With GPT-4 turbo, there were mixed results with ensemble reasoning again outperforming zero-shot chain-of-thought with self-consistency on Step 1 questions (+1.15%). In all cases, the results demonstrated improved consistency of responses with our approach. A qualitative analysis of the reasoning from the model demonstrated that the ensemble reasoning approach produces correct and helpful reasoning.
Conclusion
The proposed iterative ensemble reasoning has the potential to improve the performance of LLMs in medical question answering tasks, particularly with the less powerful LLMs like GPT-3.5 turbo and Med42-70B, which may suggest that this is a promising approach for LLMs with lower capabilities. Additionally, the findings show that our approach helps to refine the reasoning generated by the LLM and thereby improve consistency even with the more powerful GPT-4 turbo. We also identify the potential and need for human-artificial intelligence teaming to improve the reasoning beyond the limits of the model.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- College of Computing and Informatics, Drexel University, Philadelphia, PA 19104, United States.
- Department of Computer Science, University of Maryland, College Park, MD 20742, United States.
- Penn Medicine, Philadelphia, PA 19104, United States.
ORCID-Profile: Mary M Lucas
Förderung
- National Science Foundation
- Department of Defense
Zitieren
APA
Lucas, M. M., Yang, J., Pomeroy, J. K., & Yang, C. C. (2024). Reasoning with large language models for medical question answering. Journal of the American Medical Informatics Association, 31(9), 1964-1975. https://doi.org/10.1093/jamia/ocae131
Vancouver
Lucas MM, Yang J, Pomeroy JK, Yang CC. Reasoning with large language models for medical question answering. Journal of the American Medical Informatics Association. 2024;31(9):1964-1975. doi:10.1093/jamia/ocae131
BibTeX
@article{Lucas2024,
author = {Lucas, Mary M and Yang, Justin and Pomeroy, Jon K and Yang, Christopher C},
title = {Reasoning with large language models for medical question answering},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
volume = {31},
number = {9},
pages = {1964--1975},
doi = {10.1093/jamia/ocae131}
}
EUVIMED