EUVIMEDEUVIMED entdecken
Crossref

Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis

Rui Yan, Xueyuan Zhang, Zihang Jiang, Baizhi Wang, Xiuwu Bian, Fei Ren, S Kevin Zhou

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-01 · 48 (1) · 896-913

DOI: 10.1109/tpami.2025.3611531 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-01. Verfasst von Rui Yan und 6 weiteren Autor:innen. Hauptthemen laut MEDLINE: Neoplasms, Gene Expression Profiling, Image Interpretation, Computer-Assisted. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 76 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Science and Technology Innovation Key R&D Program of Chongqing, Suzhou Basic Research Program.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Integrating multimodal data of pathological image and gene expression for cancer survival analysis can achieve better results than using a single modality. However, existing multimodal learning methods ignore fine-grained interactions between both modalities, especially the interactions between biological pathways and pathological image patches. In this article, we propose a novel Pathway-Aware Multimodal Transformer (PAMT) framework for interpretable cancer survival analysis. Specifically, the PAMT learns fine-grained modality interaction through three stages: (1) In the intra-modal pathway-pathway / patch-patch interaction stage, we use the Transformer model to perform intra-modal information interaction; (2) In the inter-modal pathway-patch alignment stage, we introduce a novel label-free contrastive loss to aligns semantic information between different modalities so that the features of the two modalities are mapped to the same semantic space; and (3) In the inter-modal pathway-patch fusion stage, to model the medical prior knowledge of "genotype determines phenotype", we propose a pathway-to-patch cross fusion module to perform inter-modal information interaction under the guidance of pathway prior. In addition, the inter-modal cross fusion module of PAMT endows good interpretability, helping a pathologist to screen which pathway plays a key role, to locate where on whole slide image (WSI) are affected by the pathway, and to mine prognosis-relevant pathology image patterns. Experimental results based on three datasets of bladder urothelial carcinoma, lung squamous cell carcinoma, and lung adenocarcinoma demonstrate that the proposed framework significantly outperforms the state-of-the-art methods.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

NeoplasmsGene Expression ProfilingImage Interpretation, Computer-AssistedHumansSurvival AnalysisAlgorithmsMachine Learning

Förderung

  • National Natural Science Foundation of China
  • Science and Technology Innovation Key R&D Program of Chongqing
  • Suzhou Basic Research Program
  • Jiangsu Funding Program for Excellent Postdoctoral Talent

Zitieren

APA

Yan, R., Zhang, X., Jiang, Z., Wang, B., Bian, X., Ren, F., & Zhou, S. K. (2026). Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(1), 896-913. https://doi.org/10.1109/tpami.2025.3611531

Vancouver

Yan R, Zhang X, Jiang Z, Wang B, Bian X, Ren F, et al. Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(1):896-913. doi:10.1109/tpami.2025.3611531

BibTeX
@article{Yan2026,
  author = {Yan, Rui and Zhang, Xueyuan and Jiang, Zihang and Wang, Baizhi and Bian, Xiuwu and Ren, Fei and Zhou, S Kevin},
  title = {Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  volume = {48},
  number = {1},
  pages = {896--913},
  doi = {10.1109/tpami.2025.3611531}
}
Kostenlos starten →