EUVIMEDEUVIMED entdecken
Crossref

Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models

Jiexin Wang, Adam Jatowt, Yi Cai

ACM Transactions on the Web · 31.08.2025 · 19 (3) · 1-34

DOI: 10.1145/3723352 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on the Web, online erschienen am 22.08.2025. Verfasst von Jiexin Wang und 2 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 83 Literaturangaben. Gefördert von: Science and Technology Planning Project of Guangdong Province, National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities, South China University of Technology.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In the evolving field of Natural Language Processing (NLP), understanding the temporal context of text is increasingly critical for applications requiring advanced temporal reasoning. Traditional pre-trained language models like BERT, which rely on synchronic document collections such as BookCorpus and Wikipedia, often fall short in effectively capturing and leveraging temporal information. To address this limitation, we introduce BiTimeBERT 2.0, a novel time-aware language model pre-trained on a temporal news article collection. BiTimeBERT 2.0 incorporates temporal information through three innovative pre-training objectives: Extended Time-Aware Masked Language Modeling (ETAMLM), Document Dating (DD), and Time-Sensitive Entity Replacement (TSER). Each objective is specifically designed to target a distinct dimension of temporal information: ETAMLM enhances the model’s understanding of temporal contexts and relations, DD integrates document timestamps as explicit chronological markers, and TSER focuses on the temporal dynamics of “Person” entities. Moreover, our refined corpus preprocessing strategy reduces training time by nearly 53%, making BiTimeBERT 2.0 significantly more efficient while maintaining high performance. Experimental results show that BiTimeBERT 2.0 achieves substantial improvements across a broad range of time-related tasks and excels on datasets spanning extensive temporal ranges. These findings underscore BiTimeBERT 2.0’s potential as a powerful tool for advancing temporal reasoning in NLP. 1

Förderung

  • Science and Technology Planning Project of Guangdong Province
  • National Natural Science Foundation of China
  • Fundamental Research Funds for the Central Universities, South China University of Technology
  • Guangzhou Basic and Applied Basic Research Foundation
  • Guangdong Provincial Natural Science Foundation for Outstanding Youth Team

Zitieren

APA

Wang, J., Jatowt, A., & Cai, Y. (2025). Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models. ACM Transactions on the Web, 19(3), 1-34. https://doi.org/10.1145/3723352

Vancouver

Wang J, Jatowt A, Cai Y. Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models. ACM Transactions on the Web. 2025;19(3):1-34. doi:10.1145/3723352

BibTeX
@article{Wang2025,
  author = {Wang, Jiexin and Jatowt, Adam and Cai, Yi},
  title = {Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models},
  journal = {ACM Transactions on the Web},
  year = {2025},
  volume = {19},
  number = {3},
  pages = {1--34},
  doi = {10.1145/3723352}
}
Kostenlos starten →