A Historical Survey of Advances in Transformer Architectures
Applied Sciences · 20.05.2024 · 14 (10) · 4316
DOI: 10.3390/app14104316 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 20.05.2024. Verfasst von Ali Reza Sajun und 2 weiteren Autor:innen. Die Arbeit wurde bisher 69-mal zitiert (Crossref) und nennt 100 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: American University of Sharjah.
Abstract (Originalsprache)
In recent times, transformer-based deep learning models have risen in prominence in the field of machine learning for a variety of tasks such as computer vision and text generation. Given this increased interest, a historical outlook at the development and rapid progression of transformer-based models becomes imperative in order to gain an understanding of the rise of this key architecture. This paper presents a survey of key works related to the early development and implementation of transformer models in various domains such as generative deep learning and as backbones of large language models. Previous works are classified based on their historical approaches, followed by key works in the domain of text-based applications, image-based applications, and miscellaneous applications. A quantitative and qualitative analysis of the various approaches is presented. Additionally, recent directions of transformer-related research such as those in the biomedical and timeseries domains are discussed. Finally, future research opportunities, especially regarding the multi-modality and optimization of the transformer training process, are identified.
Förderung
- American University of Sharjah
Zitieren
APA
Sajun, A. R., Zualkernan, I., & Sankalpa, D. (2024). A Historical Survey of Advances in Transformer Architectures. Applied Sciences, 14(10), 4316. https://doi.org/10.3390/app14104316
Vancouver
Sajun AR, Zualkernan I, Sankalpa D. A Historical Survey of Advances in Transformer Architectures. Applied Sciences. 2024;14(10):4316. doi:10.3390/app14104316
BibTeX
@article{Sajun2024,
author = {Sajun, Ali Reza and Zualkernan, Imran and Sankalpa, Donthi},
title = {A Historical Survey of Advances in Transformer Architectures},
journal = {Applied Sciences},
year = {2024},
volume = {14},
number = {10},
pages = {4316},
doi = {10.3390/app14104316}
}
EUVIMED