CPT: a pre-trained unbalanced transformer for both Chinese language understanding and generation
Science China Information Sciences · 2024-05 · 67 (5)
DOI: 10.1007/s11432-021-3536-5 ↗Kurz gefasst
Fachartikel in der Zeitschrift Science China Information Sciences, online erschienen am 27.03.2024. Verfasst von Yunfan Shao und 8 weiteren Autor:innen. Die Arbeit wurde bisher 45-mal zitiert (Crossref) und nennt 45 Literaturangaben.
Abstract (Originalsprache)
In this paper, we take the advantage of previous pre-trained models (PTMs) and propose a novel Chinese pre-trained unbalanced transformer (CPT). Different from previous Chinese PTMs, CPT is designed to utilize the shared knowledge between natural language understanding (NLU) and natural language generation (NLG) to boost the performance. CPT consists of three parts: a shared encoder, an understanding decoder, and a generation decoder. Two specific decoders with a shared encoder are pre-trained with masked language modeling (MLM) and denoising auto-encoding (DAE) tasks, respectively. With the partially shared architecture and multi-task pre-training, CPT can (1) learn specific knowledge of both NLU or NLG tasks with two decoders and (2) be fine-tuned flexibly that fully exploits the potential of the model. Moreover, the unbalanced transformer saves the computational and storage cost, which makes CPT competitive and greatly accelerates the inference of text generation. Experimental results on a wide range of Chinese NLU and NLG tasks show the effectiveness of CPT.
Zitieren
APA
Shao, Y., Geng, Z., Liu, Y., Dai, J., Yan, H., Yang, F., Li, Z., Bao, H., & Qiu, X. (2024). CPT: a pre-trained unbalanced transformer for both Chinese language understanding and generation. Science China Information Sciences, 67(5). https://doi.org/10.1007/s11432-021-3536-5
Vancouver
Shao Y, Geng Z, Liu Y, Dai J, Yan H, Yang F, et al. CPT: a pre-trained unbalanced transformer for both Chinese language understanding and generation. Science China Information Sciences. 2024;67(5). doi:10.1007/s11432-021-3536-5
BibTeX
@article{Shao2024,
author = {Shao, Yunfan and Geng, Zhichao and Liu, Yitao and Dai, Junqi and Yan, Hang and Yang, Fei and Li, Zhe and Bao, Hujun and Qiu, Xipeng},
title = {CPT: a pre-trained unbalanced transformer for both Chinese language understanding and generation},
journal = {Science China Information Sciences},
year = {2024},
volume = {67},
number = {5},
doi = {10.1007/s11432-021-3536-5}
}
EUVIMED