EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

PET: Parameter-efficient Knowledge Distillation on Transformer

Hyojin Jeon iD, Seungcheol Park, Jin-Gee Kim, U Kang iD

PLOS ONE · 2023

DOI: 10.1371/journal.pone.0288060 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS ONE, 2023. Verfasst von Hyojin Jeon und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Engineering, Seoul National University, Seoul, Republic of Korea. Hauptthemen laut MEDLINE: Data Compression. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Institute of Information communications Technology Planning & Evaluation, Artificial Intelligence Graduate School Program, Artificial Intelligence Innovation Hub.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Given a large Transformer model, how can we obtain a small and computationally efficient model which maintains the performance of the original model? Transformer has shown significant performance improvements for many NLP tasks in recent years. However, their large size, expensive computational cost, and long inference time make it challenging to deploy them to resource-constrained devices. Existing Transformer compression methods mainly focus on reducing the size of the encoder ignoring the fact that the decoder takes the major portion of the long inference time. In this paper, we propose PET (P arameter -E fficient knowledge distillation on T ransformer ), an efficient Transformer compression method that reduces the size of both the encoder and decoder. In PET, we identify and exploit pairs of parameter groups for efficient weight sharing, and employ a warm-up process using a simplified task to increase the gain through Knowledge Distillation. Extensive experiments on five real-world datasets show that PET outperforms existing methods in machine translation tasks. Specifically, on the IWSLT’14 EN→DE task, PET reduces the memory usage by 81.20% and accelerates the inference speed by 45.15% compared to the uncompressed model, with a minor decrease in BLEU score of 0.27.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Data CompressionDistillationElectric Power SuppliesKnowledge

Autor:innen und Institutionen

  • Department of Computer Science and Engineering, Seoul National University, Seoul, Republic of Korea.

ORCID-Profile: Hyojin Jeon, U Kang

Förderung

  • Institute of Information communications Technology Planning & Evaluation
  • Artificial Intelligence Graduate School Program
  • Artificial Intelligence Innovation Hub

Zitieren

APA

Jeon, H., Park, S., Kim, J. G., & Kang, U. (2023). PET: Parameter-efficient Knowledge Distillation on Transformer. PLOS ONE. https://doi.org/10.1371/journal.pone.0288060

Vancouver

Jeon H, Park S, Kim JG, Kang U. PET: Parameter-efficient Knowledge Distillation on Transformer. PLOS ONE. 2023. doi:10.1371/journal.pone.0288060

BibTeX
@article{Jeon2023,
  author = {Jeon, Hyojin and Park, Seungcheol and Kim, Jin-Gee and Kang, U},
  title = {PET: Parameter-efficient Knowledge Distillation on Transformer},
  journal = {PLOS ONE},
  year = {2023},
  doi = {10.1371/journal.pone.0288060}
}
Kostenlos starten →