EUVIMEDEUVIMED entdecken
Crossref

Dynamic data-free knowledge distillation by easy-to-hard learning strategy

Jingru Li, Sheng Zhou, Liangcheng Li, Haishuai Wang, Jiajun Bu, Zhi Yu

Information Sciences · 2023-09 · 642 · 119202

DOI: 10.1016/j.ins.2023.119202 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Information Sciences, 2023-09. Verfasst von Jingru Li und 5 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 50 Literaturangaben. Gefördert von: National Natural Science Foundation of China, National Key Research and Development Program of China, Fundamental Research Funds for the Central Universities.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Data-free knowledge distillation (DFKD) is a widely-used strategy for Knowledge Distillation (KD) whose training data is not available. It trains a lightweight student model with the aid of a large pretrained teacher model without any access to training data. However, existing DFKD methods suffer from inadequate and unstable training process, as they do not adjust the generation target dynamically based on the status of the student model during learning. To address this limitation, we propose a novel DFKD method called CuDFKD. It teaches students by a dynamic strategy that gradually generates easy-to-hard pseudo samples, mirroring how humans learn. Besides, CuDFKD adapts the generation target dynamically according to the status of student model. Moreover, We provide a theoretical analysis of the majorization minimization (MM) algorithm and explain the convergence of CuDFKD. To measure the robustness and fidelity of DFKD methods, we propose two more metrics, and experiments shows CuDFKD has comparable performance to state-of-the-art (SOTA) DFKD methods on all datasets. Experiments also present that our CuDFKD has the fastest convergence and best robustness over other SOTA DFKD methods.

Förderung

  • National Natural Science Foundation of China
  • National Key Research and Development Program of China
  • Fundamental Research Funds for the Central Universities

Zitieren

APA

Li, J., Zhou, S., Li, L., Wang, H., Bu, J., & Yu, Z. (2023). Dynamic data-free knowledge distillation by easy-to-hard learning strategy. Information Sciences, 642, 119202. https://doi.org/10.1016/j.ins.2023.119202

Vancouver

Li J, Zhou S, Li L, Wang H, Bu J, Yu Z. Dynamic data-free knowledge distillation by easy-to-hard learning strategy. Information Sciences. 2023;642:119202. doi:10.1016/j.ins.2023.119202

BibTeX
@article{Li2023,
  author = {Li, Jingru and Zhou, Sheng and Li, Liangcheng and Wang, Haishuai and Bu, Jiajun and Yu, Zhi},
  title = {Dynamic data-free knowledge distillation by easy-to-hard learning strategy},
  journal = {Information Sciences},
  year = {2023},
  volume = {642},
  pages = {119202},
  doi = {10.1016/j.ins.2023.119202}
}
Kostenlos starten →