EUVIMEDEUVIMED entdecken
Crossref

UniADS: Universal Architecture-Distiller Search for Distillation Gap

Liming Lu, Zhenghan Chen, Xiaoyu Lu, Yihang Rao, Lujun Li, Shuchao Pang

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (13) · 14167-14174

DOI: 10.1609/aaai.v38i13.29327 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Liming Lu und 5 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this paper, we present UniADS, the first Universal Architecture-Distiller Search framework for co-optimizing student architecture and distillation policies. Teacher-student distillation gap limits the distillation gains. Previous approaches seek to discover the ideal student architecture while ignoring distillation settings. In UniADS, we construct a comprehensive search space encompassing an architectural search for student models, knowledge transformations in distillation strategies, distance functions, loss weights, and other vital settings. To efficiently explore the search space, we utilize the NSGA-II genetic algorithm for better crossover and mutation configurations and employ the Successive Halving algorithm for search space pruning, resulting in improved search efficiency and promising results. Extensive experiments are performed on different teacher-student pairs using CIFAR-100 and ImageNet datasets. The experimental results consistently demonstrate the superiority of our method over existing approaches. Furthermore, we provide a detailed analysis of the search results, examining the impact of each variable and extracting valuable insights and practical guidance for distillation design and implementation.

Zitieren

APA

Lu, L., Chen, Z., Lu, X., Rao, Y., Li, L., & Pang, S. (2024). UniADS: Universal Architecture-Distiller Search for Distillation Gap. Proceedings of the AAAI Conference on Artificial Intelligence, 38(13), 14167-14174. https://doi.org/10.1609/aaai.v38i13.29327

Vancouver

Lu L, Chen Z, Lu X, Rao Y, Li L, Pang S. UniADS: Universal Architecture-Distiller Search for Distillation Gap. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(13):14167-14174. doi:10.1609/aaai.v38i13.29327

BibTeX
@article{Lu2024,
  author = {Lu, Liming and Chen, Zhenghan and Lu, Xiaoyu and Rao, Yihang and Li, Lujun and Pang, Shuchao},
  title = {UniADS: Universal Architecture-Distiller Search for Distillation Gap},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {13},
  pages = {14167--14174},
  doi = {10.1609/aaai.v38i13.29327}
}
Kostenlos starten →