Multimodal Learning With Transformers: A Survey
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2023-10 · 45 (10) · 12113-12132
DOI: 10.1109/tpami.2023.3275156 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023-10. Verfasst von Peng Xu und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1011-mal zitiert (Crossref) und nennt 310 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: RAEng Research Chair, NIHR Research Professorship, NIHR Oxford Biomedical Research Centre.
Abstract (Originalsprache)
Transformer is a promising neural network learner, and has achieved great success in various machine learning tasks. Thanks to the recent prevalence of multimodal applications and Big Data, Transformer-based multimodal learning has become a hot topic in AI research. This paper presents a comprehensive survey of Transformer techniques oriented at multimodal data. The main contents of this survey include: (1) a background of multimodal learning, Transformer ecosystem, and the multimodal Big Data era, (2) a systematic review of Vanilla Transformer, Vision Transformer, and multimodal Transformers, from a geometrically topological perspective, (3) a review of multimodal Transformer applications, via two important paradigms, i.e., for multimodal pretraining and for specific multimodal tasks, (4) a summary of the common challenges and designs shared by the multimodal Transformer models and applications, and (5) a discussion of open problems and potential research directions for the community.
Förderung
- RAEng Research Chair
- NIHR Research Professorship
- NIHR Oxford Biomedical Research Centre
- InnoHK Centre for Cerebro-cardiovascular Health Engineering
- Oxford Pandemic Sciences Institute
Zitieren
APA
Xu, P., Zhu, X., & Clifton, D. A. (2023). Multimodal Learning With Transformers: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10), 12113-12132. https://doi.org/10.1109/tpami.2023.3275156
Vancouver
Xu P, Zhu X, Clifton DA. Multimodal Learning With Transformers: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(10):12113-12132. doi:10.1109/tpami.2023.3275156
BibTeX
@article{Xu2023,
author = {Xu, Peng and Zhu, Xiatian and Clifton, David A},
title = {Multimodal Learning With Transformers: A Survey},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2023},
volume = {45},
number = {10},
pages = {12113--12132},
doi = {10.1109/tpami.2023.3275156}
}
EUVIMED