Towards Understanding Evolution of Science through Language Model Series
ACM Transactions on Intelligent Systems and Technology · 31.10.2026 · 17 (5) · 1-23
DOI: 10.1145/3821210 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Intelligent Systems and Technology, online erschienen am 23.07.2026. Verfasst von Junjie Dong und 2 weiteren Autor:innen. Die Arbeit nennt 63 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, City University of Hong Kong, Hong Kong Institute for Data Science.
Abstract (Originalsprache)
We introduce AnnualBERT, a series of language models designed specifically to capture the temporal evolution of scientific text. Deviating from the prevailing paradigms of subword tokenizations and “one model to rule them all,” AnnualBERT adopts whole words as tokens and is composed of a base RoBERTa model pretrained from scratch on the full-text of 1.7 million arXiv papers published until 2008 and a collection of progressively trained models on arXiv papers at an annual basis. We demonstrate the effectiveness of AnnualBERT models by showing that they not only have comparable performance in standard tasks but also achieve state-of-the-art performance on domain-specific NLP tasks as well as link prediction tasks in the arXiv citation network. Our approach enables the pretrained models to not only improve performance on scientific text processing tasks but also to provide insights into the development of scientific discourse over time. The series of the models is available at https://huggingface.co/jd445/AnnualBERTs .
Förderung
- National Natural Science Foundation of China
- City University of Hong Kong
- Hong Kong Institute for Data Science
Zitieren
APA
Dong, J., Lyu, Z., & Ke, Q. (2026). Towards Understanding Evolution of Science through Language Model Series. ACM Transactions on Intelligent Systems and Technology, 17(5), 1-23. https://doi.org/10.1145/3821210
Vancouver
Dong J, Lyu Z, Ke Q. Towards Understanding Evolution of Science through Language Model Series. ACM Transactions on Intelligent Systems and Technology. 2026;17(5):1-23. doi:10.1145/3821210
BibTeX
@article{Dong2026,
author = {Dong, Junjie and Lyu, Zhuoqi and Ke, Qing},
title = {Towards Understanding Evolution of Science through Language Model Series},
journal = {ACM Transactions on Intelligent Systems and Technology},
year = {2026},
volume = {17},
number = {5},
pages = {1--23},
doi = {10.1145/3821210}
}
EUVIMED