EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Learning From Human Educational Wisdom: A Student-Centered Knowledge Distillation Method

Shunzhi Yang, Jinfeng Yang, MengChu Zhou, Zhenhua Huang, Wei-Shi Zheng, Xiong Yang, Jin Ren

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-06 · 46 (6) · 4188-4205

DOI: 10.1109/tpami.2024.3354928 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-06. Verfasst von Shunzhi Yang und 6 weiteren Autor:innen. Hauptthemen laut MEDLINE: Algorithms, Students. Die Arbeit wurde bisher 105-mal zitiert (Crossref) und nennt 75 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Natural Science Foundation of China, Guangdong Basic and Applied Basic Research, Shenzhen Science and Technology.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Existing studies on knowledge distillation typically focus on teacher-centered methods, in which the teacher network is trained according to its own standards before transferring the learned knowledge to a student one. However, due to differences in network structure between the teacher and the student, the knowledge learned by the former may not be desired by the latter. Inspired by human educational wisdom, this paper proposes a Student-Centered Distillation (SCD) method that enables the teacher network to adjust its knowledge transfer according to the student network's needs. We implemented SCD based on various human educational wisdom, e.g., the teacher network identified and learned the knowledge desired by the student network on the validation set, and then transferred it to the latter through the training set. To address the problems of current deficiency knowledge, hard sample learning and knowledge forgetting faced by a student network in the learning process, we introduce and improve Proportional-Integral-Derivative (PID) algorithms from automation fields to make them effective in identifying the current knowledge required by the student network. Furthermore, we propose a curriculum learning-based fuzzy strategy and apply it to the proposed PID control algorithm, such that the student network in SCD can actively pay attention to the learning of challenging samples after with certain knowledge. The overall performance of SCD is verified in multiple tasks by comparing it with state-of-the-art ones. Experimental results show that our student-centered distillation method outperforms existing teacher-centered ones.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

AlgorithmsStudentsHumansMachine LearningFuzzy LogicKnowledge

Förderung

  • National Natural Science Foundation of China
  • Guangdong Basic and Applied Basic Research
  • Shenzhen Science and Technology

Zitieren

APA

Yang, S., Yang, J., Zhou, M., Huang, Z., Zheng, W. S., Yang, X., & Ren, J. (2024). Learning From Human Educational Wisdom: A Student-Centered Knowledge Distillation Method. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(6), 4188-4205. https://doi.org/10.1109/tpami.2024.3354928

Vancouver

Yang S, Yang J, Zhou M, Huang Z, Zheng WS, Yang X, et al. Learning From Human Educational Wisdom: A Student-Centered Knowledge Distillation Method. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(6):4188-4205. doi:10.1109/tpami.2024.3354928

BibTeX
@article{Yang2024,
  author = {Yang, Shunzhi and Yang, Jinfeng and Zhou, MengChu and Huang, Zhenhua and Zheng, Wei-Shi and Yang, Xiong and Ren, Jin},
  title = {Learning From Human Educational Wisdom: A Student-Centered Knowledge Distillation Method},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {6},
  pages = {4188--4205},
  doi = {10.1109/tpami.2024.3354928}
}
Kostenlos starten →