Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models
ACM Transactions on the Web · 31.08.2025 · 19 (3) · 1-34
DOI: 10.1145/3723352 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on the Web, online erschienen am 22.08.2025. Verfasst von Jiexin Wang und 2 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 83 Literaturangaben. Gefördert von: Science and Technology Planning Project of Guangdong Province, National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities, South China University of Technology.
Abstract (Originalsprache)
In the evolving field of Natural Language Processing (NLP), understanding the temporal context of text is increasingly critical for applications requiring advanced temporal reasoning. Traditional pre-trained language models like BERT, which rely on synchronic document collections such as BookCorpus and Wikipedia, often fall short in effectively capturing and leveraging temporal information. To address this limitation, we introduce BiTimeBERT 2.0, a novel time-aware language model pre-trained on a temporal news article collection. BiTimeBERT 2.0 incorporates temporal information through three innovative pre-training objectives: Extended Time-Aware Masked Language Modeling (ETAMLM), Document Dating (DD), and Time-Sensitive Entity Replacement (TSER). Each objective is specifically designed to target a distinct dimension of temporal information: ETAMLM enhances the model’s understanding of temporal contexts and relations, DD integrates document timestamps as explicit chronological markers, and TSER focuses on the temporal dynamics of “Person” entities. Moreover, our refined corpus preprocessing strategy reduces training time by nearly 53%, making BiTimeBERT 2.0 significantly more efficient while maintaining high performance. Experimental results show that BiTimeBERT 2.0 achieves substantial improvements across a broad range of time-related tasks and excels on datasets spanning extensive temporal ranges. These findings underscore BiTimeBERT 2.0’s potential as a powerful tool for advancing temporal reasoning in NLP. 1
Förderung
- Science and Technology Planning Project of Guangdong Province
- National Natural Science Foundation of China
- Fundamental Research Funds for the Central Universities, South China University of Technology
- Guangzhou Basic and Applied Basic Research Foundation
- Guangdong Provincial Natural Science Foundation for Outstanding Youth Team
Zitieren
APA
Wang, J., Jatowt, A., & Cai, Y. (2025). Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models. ACM Transactions on the Web, 19(3), 1-34. https://doi.org/10.1145/3723352
Vancouver
Wang J, Jatowt A, Cai Y. Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models. ACM Transactions on the Web. 2025;19(3):1-34. doi:10.1145/3723352
BibTeX
@article{Wang2025,
author = {Wang, Jiexin and Jatowt, Adam and Cai, Yi},
title = {Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models},
journal = {ACM Transactions on the Web},
year = {2025},
volume = {19},
number = {3},
pages = {1--34},
doi = {10.1145/3723352}
}
EUVIMED