Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review
Progress in Biomedical Engineering · 01.04.2023 · 5 (2) · 022001
DOI: 10.1088/2516-1091/acc2fe ↗Kurz gefasst
Fachartikel in der Zeitschrift Progress in Biomedical Engineering, online erschienen am 11.04.2023. Verfasst von Can Cui und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, Vanderbilt University, Nashville, TN 37235, United States of America. Die Arbeit wurde bisher 225-mal zitiert (Crossref) und nennt 112 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: BLRD VA, CSRD VA, NIDDK NIH HHS.
Abstract (Originalsprache)
The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet complementary data that are produced during routine practice. For instance, the personalized diagnosis and treatment planning for a single cancer patient relies on various images (e.g. radiology, pathology and camera images) and non-image data (e.g. clinical data and genomic data). However, such decision-making procedures can be subjective, qualitative, and have large inter-subject variabilities. With the recent advances in multimodal deep learning technologies, an increasingly large number of efforts have been devoted to a key question: how do we extract and aggregate multimodal information to ultimately provide more objective, quantitative computer-aided clinical decision making? This paper reviews the recent studies on dealing with such a question. Briefly, this review will include the (a) overview of current multimodal learning workflows, (b) summarization of multimodal fusion methods, (c) discussion of the performance, (d) applications in disease diagnosis and prognosis, and (e) challenges and future directions.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Science, Vanderbilt University, Nashville, TN 37235, United States of America.
- Department of Pathology, Microbiology and Immunology, Vanderbilt University Medical Center, Nashville, TN 37215, United States of America.
- Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN 37215, United States of America.
- Division of Gastroenterology Hepatology, and Nutrition, Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37232, United States of America.
- Veterans Affairs Tennessee Valley Healthcare System, Nashville, TN 37212, United States of America.
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN 37235, United States of America.
ORCID-Profile: Can Cui
Förderung
- BLRD VA
- CSRD VA
- NIDDK NIH HHS
- Leona M. and Harry B. Helmsley Charitable Trust
- NSF
- CAREER
Zitieren
APA
Cui, C., Yang, H., Wang, Y., Zhao, S., Asad, Z., Coburn, L. A., Wilson, K. T., Landman, B. A., & Huo, Y. (2023). Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review. Progress in Biomedical Engineering, 5(2), 022001. https://doi.org/10.1088/2516-1091/acc2fe
Vancouver
Cui C, Yang H, Wang Y, Zhao S, Asad Z, Coburn LA, et al. Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review. Progress in Biomedical Engineering. 2023;5(2):022001. doi:10.1088/2516-1091/acc2fe
BibTeX
@article{Cui2023,
author = {Cui, Can and Yang, Haichun and Wang, Yaohong and Zhao, Shilin and Asad, Zuhayr and Coburn, Lori A and Wilson, Keith T and Landman, Bennett A and Huo, Yuankai},
title = {Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review},
journal = {Progress in Biomedical Engineering},
year = {2023},
volume = {5},
number = {2},
pages = {022001},
doi = {10.1088/2516-1091/acc2fe}
}
EUVIMED