Pre-Training a Graph Recurrent Network for Text Understanding
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-05 · 47 (5) · 4061-4074
DOI: 10.1109/tpami.2025.3538221 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-05. Verfasst von Yile Wang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 97 Literaturangaben.
Abstract (Originalsprache)
Transformer-based pre-trained models have gained much advance in recent years, Transformer architecture also becomes one of the most important backbones in natural language processing. Recent works show that the attention mechanism inside Transformer may not be necessary, and Transformer alternatives such as convolutional neural networks, multi-layer perceptron, and state space model have also been investigated. Transformer-based models have two main limitations: First, they have quadratic time complexity due to the full attention mechanism, which leads to high computational costs. Second, they rely on representation of a special token such as [CLS] to encode entire text, which limits its sentence-level expressiveness. In this paper, we consider a graph recurrent network with linear time complexity for language model pre-training, which builds a graph structure for each sequence with local token-level communications, together with a sentence-level representation detached from other normal tokens. On both English and Chinese text understanding tasks, our model can achieve comparable performance to existing pre-trained models while also achieving higher inference efficiency. Furthermore, we discovered that the representations generated by our model are more diverse and uniform compared to that of Transformer, which alleviates the problems in existing pre-trained models such as representation degradation.
Zitieren
APA
Wang, Y., Yang, L., Teng, Z., Zhou, M., & Zhang, Y. (2025). Pre-Training a Graph Recurrent Network for Text Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(5), 4061-4074. https://doi.org/10.1109/tpami.2025.3538221
Vancouver
Wang Y, Yang L, Teng Z, Zhou M, Zhang Y. Pre-Training a Graph Recurrent Network for Text Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(5):4061-4074. doi:10.1109/tpami.2025.3538221
BibTeX
@article{Wang2025,
author = {Wang, Yile and Yang, Linyi and Teng, Zhiyang and Zhou, Ming and Zhang, Yue},
title = {Pre-Training a Graph Recurrent Network for Text Understanding},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2025},
volume = {47},
number = {5},
pages = {4061--4074},
doi = {10.1109/tpami.2025.3538221}
}
EUVIMED