EUVIMEDOtkrijte EUVIMED
Crossref

An Overview of Language Models: Recent Developments and Outlook

Chengwei Wei, Yun-Cheng Wang, Bin Wang, C.-C. Jay Kuo

APSIPA Transactions on Signal and Information Processing · 2024 · 13 (2)

DOI: 10.1561/116.00000010 ↗

At a glance

Journal article in APSIPA Transactions on Signal and Information Processing, 2024. Written by Chengwei Wei and 3 co-authors. Cited 23 times so far (Crossref).

SažetakBibliografski podaciCiteRelated studies

Abstract (original language)

Language modeling studies the probability distributions over strings of texts. It is one of the most fundamental tasks in natural language processing (NLP). It has been widely used in text generation, speech recognition, machine translation, etc. Conventional language models (CLMs) aim to predict the probability of linguistic sequences in a causal manner, while pre-trained language models (PLMs) cover broader concepts and can be used in both causal sequential modeling and fine-tuning for downstream applications. PLMs have their own training paradigms (usually self-supervised) and serve as foundation models in modern NLP systems. This overview paper provides an introduction to both CLMs and PLMs from five aspects, i.e., linguistic units, architectures, training methods, evaluation methods, and applications. Furthermore, we discuss the relationship between CLMs and PLMs and shed light on the future directions of language modeling in the pre-trained era.

Cite

APA

Wei, C., Wang, Y., Wang, B., & Kuo, C. J. (2024). An Overview of Language Models: Recent Developments and Outlook. APSIPA Transactions on Signal and Information Processing, 13(2). https://doi.org/10.1561/116.00000010

Vancouver

Wei C, Wang Y, Wang B, Kuo CJ. An Overview of Language Models: Recent Developments and Outlook. APSIPA Transactions on Signal and Information Processing. 2024;13(2). doi:10.1561/116.00000010

BibTeX
@article{Wei2024,
  author = {Wei, Chengwei and Wang, Yun-Cheng and Wang, Bin and Kuo, C.-C. Jay},
  title = {An Overview of Language Models: Recent Developments and Outlook},
  journal = {APSIPA Transactions on Signal and Information Processing},
  year = {2024},
  volume = {13},
  number = {2},
  doi = {10.1561/116.00000010}
}
Start for free →