UniADS: Universal Architecture-Distiller Search for Distillation Gap
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (13) · 14167-14174
DOI: 10.1609/aaai.v38i13.29327 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Liming Lu und 5 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref).
Abstract (Originalsprache)
In this paper, we present UniADS, the first Universal Architecture-Distiller Search framework for co-optimizing student architecture and distillation policies. Teacher-student distillation gap limits the distillation gains. Previous approaches seek to discover the ideal student architecture while ignoring distillation settings. In UniADS, we construct a comprehensive search space encompassing an architectural search for student models, knowledge transformations in distillation strategies, distance functions, loss weights, and other vital settings. To efficiently explore the search space, we utilize the NSGA-II genetic algorithm for better crossover and mutation configurations and employ the Successive Halving algorithm for search space pruning, resulting in improved search efficiency and promising results. Extensive experiments are performed on different teacher-student pairs using CIFAR-100 and ImageNet datasets. The experimental results consistently demonstrate the superiority of our method over existing approaches. Furthermore, we provide a detailed analysis of the search results, examining the impact of each variable and extracting valuable insights and practical guidance for distillation design and implementation.
Zitieren
APA
Lu, L., Chen, Z., Lu, X., Rao, Y., Li, L., & Pang, S. (2024). UniADS: Universal Architecture-Distiller Search for Distillation Gap. Proceedings of the AAAI Conference on Artificial Intelligence, 38(13), 14167-14174. https://doi.org/10.1609/aaai.v38i13.29327
Vancouver
Lu L, Chen Z, Lu X, Rao Y, Li L, Pang S. UniADS: Universal Architecture-Distiller Search for Distillation Gap. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(13):14167-14174. doi:10.1609/aaai.v38i13.29327
BibTeX
@article{Lu2024,
author = {Lu, Liming and Chen, Zhenghan and Lu, Xiaoyu and Rao, Yihang and Li, Lujun and Pang, Shuchao},
title = {UniADS: Universal Architecture-Distiller Search for Distillation Gap},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {13},
pages = {14167--14174},
doi = {10.1609/aaai.v38i13.29327}
}
EUVIMED