EUVIMEDEUVIMED entdecken
Crossref

Survival Prediction via Hierarchical Multimodal Co-Attention Transformer: A Computational Histology-Radiology Solution

Zhe Li, Yuming Jiang, Mengkang Lu, Ruijiang Li, Yong Xia

IEEE Transactions on Medical Imaging · 2023-09 · 42 (9) · 2678-2689

DOI: 10.1109/tmi.2023.3263010 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Medical Imaging, 2023-09. Verfasst von Zhe Li und 4 weiteren Autor:innen. Hauptthemen laut MEDLINE: Radiology. Die Arbeit wurde bisher 88-mal zitiert (Crossref) und nennt 45 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, Key Research and Development Program of Shaanxi Province, China.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The rapid advances in deep learning-based computational pathology and radiology have demonstrated the promise of using whole slide images (WSIs) and radiology images for survival prediction in cancer patients. However, most image-based survival prediction methods are limited to using either histology or radiology alone, leaving integrated approaches across histology and radiology relatively underdeveloped. There are two main challenges in integrating WSIs and radiology images: (1) the gigapixel nature of WSIs and (2) the vast difference in spatial scales between WSIs and radiology images. To address these challenges, in this work, we propose an interpretable, weakly-supervised, multimodal learning framework, called Hierarchical Multimodal Co-Attention Transformer (HMCAT), to integrate WSIs and radiology images for survival prediction. Our approach first uses hierarchical feature extractors to capture various information including cellular features, cellular organization, and tissue phenotypes in WSIs. Then the hierarchical radiology-guided co- attention (HRCA) in HMCAT characterizes the multimodal interactions between hierarchical histology-based visual concepts and radiology features and learns hierarchical co- attention mappings for two modalities. Finally, HMCAT combines their complementary information into a multimodal risk score and discovers prognostic features from two modalities by multimodal interpretability. We apply our approach to two cancer datasets (365 WSIs with matched magnetic resonance [MR] images and 213 WSIs with matched computed tomography [CT] images). Our results demonstrate that the proposed HMCAT consistently achieves superior performance over the unimodal approaches trained on either histology or radiology data alone, as well as other state-of-the-art methods.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

RadiologyRadiographyTomography, X-Ray ComputedHistological Techniques

Förderung

  • National Key Research and Development Program of China
  • National Natural Science Foundation of China
  • Key Research and Development Program of Shaanxi Province, China

Zitieren

APA

Li, Z., Jiang, Y., Lu, M., Li, R., & Xia, Y. (2023). Survival Prediction via Hierarchical Multimodal Co-Attention Transformer: A Computational Histology-Radiology Solution. IEEE Transactions on Medical Imaging, 42(9), 2678-2689. https://doi.org/10.1109/tmi.2023.3263010

Vancouver

Li Z, Jiang Y, Lu M, Li R, Xia Y. Survival Prediction via Hierarchical Multimodal Co-Attention Transformer: A Computational Histology-Radiology Solution. IEEE Transactions on Medical Imaging. 2023;42(9):2678-2689. doi:10.1109/tmi.2023.3263010

BibTeX
@article{Li2023,
  author = {Li, Zhe and Jiang, Yuming and Lu, Mengkang and Li, Ruijiang and Xia, Yong},
  title = {Survival Prediction via Hierarchical Multimodal Co-Attention Transformer: A Computational Histology-Radiology Solution},
  journal = {IEEE Transactions on Medical Imaging},
  year = {2023},
  volume = {42},
  number = {9},
  pages = {2678--2689},
  doi = {10.1109/tmi.2023.3263010}
}
Kostenlos starten →