EUVIMEDEUVIMED entdecken
Systematische ÜbersichtOpen Access · CC BY 4.0

A Systematic Review of Transformer-Based Pre-Trained Language Models through Self-Supervised Learning

Evans Kotei, Ramkumar Thirunavukarasu

Information · 16.03.2023 · 14 (3) · 187

DOI: 10.3390/info14030187 ↗

Kurz gefasst

Systematische Übersicht in der Zeitschrift Information, online erschienen am 16.03.2023. Verfasst von Evans Kotei und 1 weiteren Autor:innen. Die Arbeit wurde bisher 88-mal zitiert (Crossref) und nennt 126 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Transfer learning is a technique utilized in deep learning applications to transmit learned inference to a different target domain. The approach is mainly to solve the problem of a few training datasets resulting in model overfitting, which affects model performance. The study was carried out on publications retrieved from various digital libraries such as SCOPUS, ScienceDirect, IEEE Xplore, ACM Digital Library, and Google Scholar, which formed the Primary studies. Secondary studies were retrieved from Primary articles using the backward and forward snowballing approach. Based on set inclusion and exclusion parameters, relevant publications were selected for review. The study focused on transfer learning pretrained NLP models based on the deep transformer network. BERT and GPT were the two elite pretrained models trained to classify global and local representations based on larger unlabeled text datasets through self-supervised learning. Pretrained transformer models offer numerous advantages to natural language processing models, such as knowledge transfer to downstream tasks that deal with drawbacks associated with training a model from scratch. This review gives a comprehensive view of transformer architecture, self-supervised learning and pretraining concepts in language models, and their adaptation to downstream tasks. Finally, we present future directions to further improvement in pretrained transformer-based language models.

Zitieren

APA

Kotei, E., & Thirunavukarasu, R. (2023). A Systematic Review of Transformer-Based Pre-Trained Language Models through Self-Supervised Learning. Information, 14(3), 187. https://doi.org/10.3390/info14030187

Vancouver

Kotei E, Thirunavukarasu R. A Systematic Review of Transformer-Based Pre-Trained Language Models through Self-Supervised Learning. Information. 2023;14(3):187. doi:10.3390/info14030187

BibTeX
@article{Kotei2023,
  author = {Kotei, Evans and Thirunavukarasu, Ramkumar},
  title = {A Systematic Review of Transformer-Based Pre-Trained Language Models through Self-Supervised Learning},
  journal = {Information},
  year = {2023},
  volume = {14},
  number = {3},
  pages = {187},
  doi = {10.3390/info14030187}
}
Kostenlos starten →