EUVIMEDEUVIMED entdecken
Open Access

Reasoning with large language models for medical question answering

Mary M Lucas iD, Justin Yang, Jon K Pomeroy, Christopher C Yang

Journal of the American Medical Informatics Association · 01.09.2024 · 31 (9) · 1964-1975

DOI: 10.1093/jamia/ocae131 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 03.07.2024. Verfasst von Mary M Lucas und 3 weiteren Autor:innen. Beteiligte Einrichtung: College of Computing and Informatics, Drexel University, Philadelphia, PA 19104, United States. Hauptthemen laut MEDLINE: Natural Language Processing. Die Arbeit wurde bisher 76-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: National Science Foundation, Department of Defense.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

To investigate approaches of reasoning with large language models (LLMs) and to propose a new prompting approach, ensemble reasoning, to improve medical question answering performance with refined reasoning and reduced inconsistency. Materials and Methods We used multiple choice questions from the USMLE Sample Exam question files on 2 closed-source commercial and 1 open-source clinical LLM to evaluate our proposed approach ensemble reasoning.

Results

On GPT-3.5 turbo and Med42-70B, our proposed ensemble reasoning approach outperformed zero-shot chain-of-thought with self-consistency on Steps 1, 2, and 3 questions (+3.44%, +4.00%, and +2.54%) and (2.3%, 5.00%, and 4.15%), respectively. With GPT-4 turbo, there were mixed results with ensemble reasoning again outperforming zero-shot chain-of-thought with self-consistency on Step 1 questions (+1.15%). In all cases, the results demonstrated improved consistency of responses with our approach. A qualitative analysis of the reasoning from the model demonstrated that the ensemble reasoning approach produces correct and helpful reasoning.

Conclusion

The proposed iterative ensemble reasoning has the potential to improve the performance of LLMs in medical question answering tasks, particularly with the less powerful LLMs like GPT-3.5 turbo and Med42-70B, which may suggest that this is a promising approach for LLMs with lower capabilities. Additionally, the findings show that our approach helps to refine the reasoning generated by the LLM and thereby improve consistency even with the more powerful GPT-4 turbo. We also identify the potential and need for human-artificial intelligence teaming to improve the reasoning beyond the limits of the model.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingHumansEducational Measurement

Stichwörter der Autor:innen

artificial intelligenceclinical reasoninglarge language modelmachine reasoning

Autor:innen und Institutionen

  • College of Computing and Informatics, Drexel University, Philadelphia, PA 19104, United States.
  • Department of Computer Science, University of Maryland, College Park, MD 20742, United States.
  • Penn Medicine, Philadelphia, PA 19104, United States.

ORCID-Profile: Mary M Lucas

Förderung

  • National Science Foundation
  • Department of Defense

Zitieren

APA

Lucas, M. M., Yang, J., Pomeroy, J. K., & Yang, C. C. (2024). Reasoning with large language models for medical question answering. Journal of the American Medical Informatics Association, 31(9), 1964-1975. https://doi.org/10.1093/jamia/ocae131

Vancouver

Lucas MM, Yang J, Pomeroy JK, Yang CC. Reasoning with large language models for medical question answering. Journal of the American Medical Informatics Association. 2024;31(9):1964-1975. doi:10.1093/jamia/ocae131

BibTeX
@article{Lucas2024,
  author = {Lucas, Mary M and Yang, Justin and Pomeroy, Jon K and Yang, Christopher C},
  title = {Reasoning with large language models for medical question answering},
  journal = {Journal of the American Medical Informatics Association},
  year = {2024},
  volume = {31},
  number = {9},
  pages = {1964--1975},
  doi = {10.1093/jamia/ocae131}
}
Kostenlos starten →