Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-01 · 48 (1) · 896-913
DOI: 10.1109/tpami.2025.3611531 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-01. Verfasst von Rui Yan und 6 weiteren Autor:innen. Hauptthemen laut MEDLINE: Neoplasms, Gene Expression Profiling, Image Interpretation, Computer-Assisted. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 76 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Science and Technology Innovation Key R&D Program of Chongqing, Suzhou Basic Research Program.
Abstract (Originalsprache)
Integrating multimodal data of pathological image and gene expression for cancer survival analysis can achieve better results than using a single modality. However, existing multimodal learning methods ignore fine-grained interactions between both modalities, especially the interactions between biological pathways and pathological image patches. In this article, we propose a novel Pathway-Aware Multimodal Transformer (PAMT) framework for interpretable cancer survival analysis. Specifically, the PAMT learns fine-grained modality interaction through three stages: (1) In the intra-modal pathway-pathway / patch-patch interaction stage, we use the Transformer model to perform intra-modal information interaction; (2) In the inter-modal pathway-patch alignment stage, we introduce a novel label-free contrastive loss to aligns semantic information between different modalities so that the features of the two modalities are mapped to the same semantic space; and (3) In the inter-modal pathway-patch fusion stage, to model the medical prior knowledge of "genotype determines phenotype", we propose a pathway-to-patch cross fusion module to perform inter-modal information interaction under the guidance of pathway prior. In addition, the inter-modal cross fusion module of PAMT endows good interpretability, helping a pathologist to screen which pathway plays a key role, to locate where on whole slide image (WSI) are affected by the pathway, and to mine prognosis-relevant pathology image patterns. Experimental results based on three datasets of bladder urothelial carcinoma, lung squamous cell carcinoma, and lung adenocarcinoma demonstrate that the proposed framework significantly outperforms the state-of-the-art methods.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Förderung
- National Natural Science Foundation of China
- Science and Technology Innovation Key R&D Program of Chongqing
- Suzhou Basic Research Program
- Jiangsu Funding Program for Excellent Postdoctoral Talent
Zitieren
APA
Yan, R., Zhang, X., Jiang, Z., Wang, B., Bian, X., Ren, F., & Zhou, S. K. (2026). Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(1), 896-913. https://doi.org/10.1109/tpami.2025.3611531
Vancouver
Yan R, Zhang X, Jiang Z, Wang B, Bian X, Ren F, et al. Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(1):896-913. doi:10.1109/tpami.2025.3611531
BibTeX
@article{Yan2026,
author = {Yan, Rui and Zhang, Xueyuan and Jiang, Zihang and Wang, Baizhi and Bian, Xiuwu and Ren, Fei and Zhou, S Kevin},
title = {Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2026},
volume = {48},
number = {1},
pages = {896--913},
doi = {10.1109/tpami.2025.3611531}
}
EUVIMED