EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Historical Survey of Advances in Transformer Architectures

Ali Reza Sajun, Imran Zualkernan, Donthi Sankalpa

Applied Sciences · 20.05.2024 · 14 (10) · 4316

DOI: 10.3390/app14104316 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 20.05.2024. Verfasst von Ali Reza Sajun und 2 weiteren Autor:innen. Die Arbeit wurde bisher 69-mal zitiert (Crossref) und nennt 100 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: American University of Sharjah.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In recent times, transformer-based deep learning models have risen in prominence in the field of machine learning for a variety of tasks such as computer vision and text generation. Given this increased interest, a historical outlook at the development and rapid progression of transformer-based models becomes imperative in order to gain an understanding of the rise of this key architecture. This paper presents a survey of key works related to the early development and implementation of transformer models in various domains such as generative deep learning and as backbones of large language models. Previous works are classified based on their historical approaches, followed by key works in the domain of text-based applications, image-based applications, and miscellaneous applications. A quantitative and qualitative analysis of the various approaches is presented. Additionally, recent directions of transformer-related research such as those in the biomedical and timeseries domains are discussed. Finally, future research opportunities, especially regarding the multi-modality and optimization of the transformer training process, are identified.

Förderung

  • American University of Sharjah

Zitieren

APA

Sajun, A. R., Zualkernan, I., & Sankalpa, D. (2024). A Historical Survey of Advances in Transformer Architectures. Applied Sciences, 14(10), 4316. https://doi.org/10.3390/app14104316

Vancouver

Sajun AR, Zualkernan I, Sankalpa D. A Historical Survey of Advances in Transformer Architectures. Applied Sciences. 2024;14(10):4316. doi:10.3390/app14104316

BibTeX
@article{Sajun2024,
  author = {Sajun, Ali Reza and Zualkernan, Imran and Sankalpa, Donthi},
  title = {A Historical Survey of Advances in Transformer Architectures},
  journal = {Applied Sciences},
  year = {2024},
  volume = {14},
  number = {10},
  pages = {4316},
  doi = {10.3390/app14104316}
}
Kostenlos starten →