Sequence-aware Knowledge Distillation for a Lightweight Event Representation
ACM Transactions on Information Systems · 30.04.2023 · 41 (2) · 1-30
DOI: 10.1145/3545798 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Information Systems, online erschienen am 21.12.2022. Verfasst von Jianming Zheng und 3 weiteren Autor:innen. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 54 Literaturangaben.
Abstract (Originalsprache)
Event representation targets to model the event-reasoning process as a machine-readable format. Previous studies on event representation mostly concentrate on a sole modeling perspective and have not well investigated the scenario-level knowledge, which can cause information loss. To cope with this dilemma, we propose a unified fine-tuning architecture-based approach ( UniFA-S ) that integrates all levels of trainings, including the scenario-level knowledge. However, another challenge for existing models is the ever-increasing computation overheads, restricting the deployment ability on limited resources devices. Hence, in this article, we aim to compress the cumbersome model UniFA-S into a lighter and easy-to-deploy one without much performance damage. To this end, we propose a sequence-aware knowledge distillation model (SaKD) that employs a dynamic self-distillation on the decouple-compress-couple framework for compressing UniFA-S , which cannot only realize the model compression, but also retain the integrity of individual components. We also design two fitting strategies to address the less-supervised issue at the distillation stage. Comprehensive experiments on representation-and-inference ability-based tasks validate the effectiveness of SaKD. Compared to UniFA-S , SaKD realizes a more portable event representation model at the cost of only 1.0% performance drop in terms of accuracy or Spearman’s correlation, which is far less than other knowledge distillation models.
Zitieren
APA
Zheng, J., Cai, F., Ling, Y., & Chen, H. (2023). Sequence-aware Knowledge Distillation for a Lightweight Event Representation. ACM Transactions on Information Systems, 41(2), 1-30. https://doi.org/10.1145/3545798
Vancouver
Zheng J, Cai F, Ling Y, Chen H. Sequence-aware Knowledge Distillation for a Lightweight Event Representation. ACM Transactions on Information Systems. 2023;41(2):1-30. doi:10.1145/3545798
BibTeX
@article{Zheng2023,
author = {Zheng, Jianming and Cai, Fei and Ling, Yanxiang and Chen, Honghui},
title = {Sequence-aware Knowledge Distillation for a Lightweight Event Representation},
journal = {ACM Transactions on Information Systems},
year = {2023},
volume = {41},
number = {2},
pages = {1--30},
doi = {10.1145/3545798}
}
EUVIMED