EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Towards Understanding Evolution of Science through Language Model Series

Junjie Dong, Zhuoqi Lyu, Qing Ke

ACM Transactions on Intelligent Systems and Technology · 31.10.2026 · 17 (5) · 1-23

DOI: 10.1145/3821210 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Intelligent Systems and Technology, online erschienen am 23.07.2026. Verfasst von Junjie Dong und 2 weiteren Autor:innen. Die Arbeit nennt 63 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, City University of Hong Kong, Hong Kong Institute for Data Science.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We introduce AnnualBERT, a series of language models designed specifically to capture the temporal evolution of scientific text. Deviating from the prevailing paradigms of subword tokenizations and “one model to rule them all,” AnnualBERT adopts whole words as tokens and is composed of a base RoBERTa model pretrained from scratch on the full-text of 1.7 million arXiv papers published until 2008 and a collection of progressively trained models on arXiv papers at an annual basis. We demonstrate the effectiveness of AnnualBERT models by showing that they not only have comparable performance in standard tasks but also achieve state-of-the-art performance on domain-specific NLP tasks as well as link prediction tasks in the arXiv citation network. Our approach enables the pretrained models to not only improve performance on scientific text processing tasks but also to provide insights into the development of scientific discourse over time. The series of the models is available at https://huggingface.co/jd445/AnnualBERTs .

Förderung

  • National Natural Science Foundation of China
  • City University of Hong Kong
  • Hong Kong Institute for Data Science

Zitieren

APA

Dong, J., Lyu, Z., & Ke, Q. (2026). Towards Understanding Evolution of Science through Language Model Series. ACM Transactions on Intelligent Systems and Technology, 17(5), 1-23. https://doi.org/10.1145/3821210

Vancouver

Dong J, Lyu Z, Ke Q. Towards Understanding Evolution of Science through Language Model Series. ACM Transactions on Intelligent Systems and Technology. 2026;17(5):1-23. doi:10.1145/3821210

BibTeX
@article{Dong2026,
  author = {Dong, Junjie and Lyu, Zhuoqi and Ke, Qing},
  title = {Towards Understanding Evolution of Science through Language Model Series},
  journal = {ACM Transactions on Intelligent Systems and Technology},
  year = {2026},
  volume = {17},
  number = {5},
  pages = {1--23},
  doi = {10.1145/3821210}
}
Kostenlos starten →