EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Multimodal Learning With Transformers: A Survey

Peng Xu, Xiatian Zhu, David A Clifton

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2023-10 · 45 (10) · 12113-12132

DOI: 10.1109/tpami.2023.3275156 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023-10. Verfasst von Peng Xu und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1011-mal zitiert (Crossref) und nennt 310 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: RAEng Research Chair, NIHR Research Professorship, NIHR Oxford Biomedical Research Centre.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Transformer is a promising neural network learner, and has achieved great success in various machine learning tasks. Thanks to the recent prevalence of multimodal applications and Big Data, Transformer-based multimodal learning has become a hot topic in AI research. This paper presents a comprehensive survey of Transformer techniques oriented at multimodal data. The main contents of this survey include: (1) a background of multimodal learning, Transformer ecosystem, and the multimodal Big Data era, (2) a systematic review of Vanilla Transformer, Vision Transformer, and multimodal Transformers, from a geometrically topological perspective, (3) a review of multimodal Transformer applications, via two important paradigms, i.e., for multimodal pretraining and for specific multimodal tasks, (4) a summary of the common challenges and designs shared by the multimodal Transformer models and applications, and (5) a discussion of open problems and potential research directions for the community.

Förderung

  • RAEng Research Chair
  • NIHR Research Professorship
  • NIHR Oxford Biomedical Research Centre
  • InnoHK Centre for Cerebro-cardiovascular Health Engineering
  • Oxford Pandemic Sciences Institute

Zitieren

APA

Xu, P., Zhu, X., & Clifton, D. A. (2023). Multimodal Learning With Transformers: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10), 12113-12132. https://doi.org/10.1109/tpami.2023.3275156

Vancouver

Xu P, Zhu X, Clifton DA. Multimodal Learning With Transformers: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(10):12113-12132. doi:10.1109/tpami.2023.3275156

BibTeX
@article{Xu2023,
  author = {Xu, Peng and Zhu, Xiatian and Clifton, David A},
  title = {Multimodal Learning With Transformers: A Survey},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2023},
  volume = {45},
  number = {10},
  pages = {12113--12132},
  doi = {10.1109/tpami.2023.3275156}
}
Kostenlos starten →