MMsurv: a multimodal multi-instance multi-cancer survival prediction model integrating pathological images, clinical information, and sequencing data
Briefings in Bioinformatics · 01.05.2025 · 26 (3)
DOI: 10.1093/bib/bbaf209 ↗Kurz gefasst
Fachartikel in der Zeitschrift Briefings in Bioinformatics, online erschienen am 14.05.2025. Verfasst von Hailong Yang und 8 weiteren Autor:innen. Beteiligte Einrichtung: School of Electrical and Information Engineering, Anhui University of Technology, No. 1530 Maxiang Road, Huashan District, Ma'anshan, Anhui 243032, C…. Hauptthemen laut MEDLINE: Neoplasms, Deep Learning. Die Arbeit wurde bisher 27-mal zitiert (Crossref) und nennt 48 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: National Natural Science Foundation of China, Hainan Provincial Natural Science Foundation of China, Natural Science Foundation of Hubei Province.
Abstract (Originalsprache)
Accurate prediction of patient survival rates in cancer treatment is essential for effective therapeutic planning. Unfortunately, current models often underutilize the extensive multimodal data available, affecting confidence in predictions. This study presents MMSurv, an interpretable multimodal deep learning model to predict survival in different types of cancer. MMSurv integrates clinical information, sequencing data, and hematoxylin and eosin-stained whole-slide images (WSIs) to forecast patient survival. Specifically, we segment tumor regions from WSIs into image tiles and employ neural networks to encode each tile into one-dimensional feature vectors. We then optimize clinical features by applying word embedding techniques, inspired by natural language processing, to the clinical data. To better utilize the complementarity of multimodal data, this study proposes a novel fusion method, multimodal fusion method based on compact bilinear pooling and transformer, which integrates bilinear pooling with Transformer architecture. The fused features are then processed through a dual-layer multi-instance learning model to remove prognosis-irrelevant image patches and predict each patient's survival risk. Furthermore, we employ cell segmentation to investigate the cellular composition within the tiles that received high attention from the model, thereby enhancing its interpretive capacity. We evaluate our approach on six cancer types from The Cancer Genome Atlas. The results demonstrate that utilizing multimodal data leads to higher predictive accuracy compared to using single-modal image data, with an average C-index increase from 0.6750 to 0.7283. Additionally, we compare our proposed baseline model with state-of-the-art methods using the C-index and five-fold cross-validation approach, revealing a significant average improvement of nearly 10% in our model's performance.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Electrical and Information Engineering, Anhui University of Technology, No. 1530 Maxiang Road, Huashan District, Ma'anshan, Anhui 243032, China.
- Department of Sciences, Geneis Beijing Co., Ltd., No. 31 Xinbei Road, Laiguangying, Chaoyang District, Beijing 100102, China.
- Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education/Beijing), Department of Thoracic Surgery II, Peking University Cancer Hospital & Institute, No. 52 Fucheng Road, Haidian District, Beiji…
- Department of Pathology, Beijing Friendship Hospital, Capital Medical University, No. 95 Yong'an Road, Xicheng District, Beijing 100050, China.
- Qinghe Clinic, Beijing North Medical District, Chinese PLA General Hospital, No. 100 Youyi Road, Haidian District, Beijing 100080, China.
- School of Mathematics and Statistics, Hainan Normal University, No. 99 Longkunnan Road, Longhua District, Haikou, Hainan 571158, China.
ORCID-Profile: Hailong Yang, Jialiang Yang
Förderung
- National Natural Science Foundation of China
- Hainan Provincial Natural Science Foundation of China
- Natural Science Foundation of Hubei Province
Zitieren
APA
Yang, H., Wang, J., Wang, W., Shi, S., Liu, L., Yao, Y., Tian, G., Wang, P., & Yang, J. (2025). MMsurv: a multimodal multi-instance multi-cancer survival prediction model integrating pathological images, clinical information, and sequencing data. Briefings in Bioinformatics, 26(3). https://doi.org/10.1093/bib/bbaf209
Vancouver
Yang H, Wang J, Wang W, Shi S, Liu L, Yao Y, et al. MMsurv: a multimodal multi-instance multi-cancer survival prediction model integrating pathological images, clinical information, and sequencing data. Briefings in Bioinformatics. 2025;26(3). doi:10.1093/bib/bbaf209
BibTeX
@article{Yang2025,
author = {Yang, Hailong and Wang, Jia and Wang, Wenyan and Shi, Shufang and Liu, Lijing and Yao, Yuhua and Tian, Geng and Wang, Peizhen and Yang, Jialiang},
title = {MMsurv: a multimodal multi-instance multi-cancer survival prediction model integrating pathological images, clinical information, and sequencing data},
journal = {Briefings in Bioinformatics},
year = {2025},
volume = {26},
number = {3},
doi = {10.1093/bib/bbaf209}
}
EUVIMED