Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook
Journal of Medical Internet Research · 2024-09-25 · 26 · e59505
DOI: 10.2196/59505 ↗At a glance
Journal article in Journal of Medical Internet Research, published online on 2024-09-25. Written by Rawan AlSaad and 6 co-authors. Institution: Weill Cornell Medicine-Qatar, Education City, Doha, Qatar. Major MEDLINE topics: Delivery of Health Care. Cited 391 times so far (Crossref); 199 references. The full text is freely available in PubMed Central.
Abstract (original language)
In the complex and multidimensional field of medicine, multimodal data are prevalent and crucial for informed clinical decisions. Multimodal data span a broad spectrum of data types, including medical images (eg, MRI and CT scans), time-series data (eg, sensor data from wearable devices and electronic health records), audio recordings (eg, heart and respiratory sounds and patient interviews), text (eg, clinical notes and research articles), videos (eg, surgical procedures), and omics data (eg, genomics and proteomics). While advancements in large language models (LLMs) have enabled new applications for knowledge retrieval and processing in the medical field, most LLMs remain limited to processing unimodal data, typically text-based content, and often overlook the importance of integrating the diverse data modalities encountered in clinical practice. This paper aims to present a detailed, practical, and solution-oriented perspective on the use of multimodal LLMs (M-LLMs) in the medical field. Our investigation spanned M-LLM foundational principles, current and potential applications, technical and ethical challenges, and future research directions. By connecting these elements, we aimed to provide a comprehensive framework that links diverse aspects of M-LLMs, offering a unified vision for their future in health care. This approach aims to guide both future research and practical implementations of M-LLMs in health care, positioning them as a paradigm shift toward integrated, multimodal data–driven medical practice. We anticipate that this work will spark further discussion and inspire the development of innovative approaches in the next generation of medical M-LLM systems.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Weill Cornell Medicine-Qatar, Education City, Doha, Qatar.
- Qatar Computing Research Institute, Hamad Bin Khalifa University, Doha, Qatar.
- Department of Computer Science and Software Engineering, United Arab Emirates University, Al Ain, United Arab Emirates.
ORCID profiles: Rawan AlSaad, Alaa Abd-Alrazaq, Sabri Boughorbel, Arfan Ahmed, Max-Antoine Renault, Rafat Damseh, Javaid Sheikh
Cite
APA
AlSaad, R., Abd-Alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M. A., Damseh, R., & Sheikh, J. (2024). Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. Journal of Medical Internet Research, 26, e59505. https://doi.org/10.2196/59505
Vancouver
AlSaad R, Abd-Alrazaq A, Boughorbel S, Ahmed A, Renault MA, Damseh R, et al. Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. Journal of Medical Internet Research. 2024;26:e59505. doi:10.2196/59505
BibTeX
@article{AlSaad2024,
author = {AlSaad, Rawan and Abd-Alrazaq, Alaa and Boughorbel, Sabri and Ahmed, Arfan and Renault, Max-Antoine and Damseh, Rafat and Sheikh, Javaid},
title = {Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook},
journal = {Journal of Medical Internet Research},
year = {2024},
volume = {26},
pages = {e59505},
doi = {10.2196/59505}
}
EUVIMED