EUVIMEDEUVIMED entdecken
Crossref

Pre-Training a Graph Recurrent Network for Text Understanding

Yile Wang, Linyi Yang, Zhiyang Teng, Ming Zhou, Yue Zhang

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-05 · 47 (5) · 4061-4074

DOI: 10.1109/tpami.2025.3538221 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-05. Verfasst von Yile Wang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 97 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Transformer-based pre-trained models have gained much advance in recent years, Transformer architecture also becomes one of the most important backbones in natural language processing. Recent works show that the attention mechanism inside Transformer may not be necessary, and Transformer alternatives such as convolutional neural networks, multi-layer perceptron, and state space model have also been investigated. Transformer-based models have two main limitations: First, they have quadratic time complexity due to the full attention mechanism, which leads to high computational costs. Second, they rely on representation of a special token such as [CLS] to encode entire text, which limits its sentence-level expressiveness. In this paper, we consider a graph recurrent network with linear time complexity for language model pre-training, which builds a graph structure for each sequence with local token-level communications, together with a sentence-level representation detached from other normal tokens. On both English and Chinese text understanding tasks, our model can achieve comparable performance to existing pre-trained models while also achieving higher inference efficiency. Furthermore, we discovered that the representations generated by our model are more diverse and uniform compared to that of Transformer, which alleviates the problems in existing pre-trained models such as representation degradation.

Zitieren

APA

Wang, Y., Yang, L., Teng, Z., Zhou, M., & Zhang, Y. (2025). Pre-Training a Graph Recurrent Network for Text Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(5), 4061-4074. https://doi.org/10.1109/tpami.2025.3538221

Vancouver

Wang Y, Yang L, Teng Z, Zhou M, Zhang Y. Pre-Training a Graph Recurrent Network for Text Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(5):4061-4074. doi:10.1109/tpami.2025.3538221

BibTeX
@article{Wang2025,
  author = {Wang, Yile and Yang, Linyi and Teng, Zhiyang and Zhou, Ming and Zhang, Yue},
  title = {Pre-Training a Graph Recurrent Network for Text Understanding},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {5},
  pages = {4061--4074},
  doi = {10.1109/tpami.2025.3538221}
}
Kostenlos starten →