EUVIMEDEUVIMED entdecken
Crossref

Allies Teach Better Than Enemies: Inverse Adversaries for Robust Knowledge Distillation

Junhao Dong, Raoof Zare Moayedi, Yew-Soon Ong, Seyed-Mohsen Moosavi-Dezfooli

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-06 · 48 (6) · 6557-6569

DOI: 10.1109/tpami.2026.3660863 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-06. Verfasst von Junhao Dong und 3 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 55 Literaturangaben. Gefördert von: National Research Foundation, Singapore and Infocomm Media Development Authority, Centre for Frontier Artificial Intelligence Research.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Adversarially robust knowledge distillation aims to compress a large-scale robust teacher model into a lightweight student counterpart while preserving adversarial robustness and natural performance. Previous methods primarily focused on aligning knowledge (e.g., predictions) between teacher and student models to transfer robustness. However, potentially incorrect predictions from the teacher can misguide the student, negatively impacting robustness transfer. To circumvent this, we propose a novel adversarially robust knowledge distillation scheme that promotes alignment towards more benign predictions rather than incorrect ones by refining inputs into so-called "inverse adversarial examples" via simply reversing the sign of adversarial perturbation. Through a comprehensive investigation of the properties of inverse adversaries, we provide new theoretical insights showing how mimicking the behavior of the teacher model on inverse adversaries facilitates reliable robustness transfer built upon the implicit connection between robustness and the input gradient information. We thus design a gradient matching mechanism between teacher and student models utilizing inverse adversaries to facilitate robust knowledge alignment. Furthermore, inspired by our analysis of the correlation between robustness and adversarial transferability, we propose a weight-space disruption strategy that jointly interacts with both teacher and student models to find a shared direction for better robustness transfer. Empirical evaluations across various datasets demonstrate that our method achieves state-of-the-art robustness and natural performance. Notably, on ImageNet, our approach outperforms prior methods by approximately 3.8% in both clean and robust accuracy. Moreover, we show that incorporating auxiliary generated data into distillation further boosts robustness. Our method can also be generalized to multimodal architectures.

Förderung

  • National Research Foundation
  • Singapore and Infocomm Media Development Authority
  • Centre for Frontier Artificial Intelligence Research
  • Institute of High Performance Computing
  • A*Star
  • College of Computing and Data Science at Nanyang Technological University

Zitieren

APA

Dong, J., Moayedi, R. Z., Ong, Y. S., & Moosavi-Dezfooli, S. M. (2026). Allies Teach Better Than Enemies: Inverse Adversaries for Robust Knowledge Distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(6), 6557-6569. https://doi.org/10.1109/tpami.2026.3660863

Vancouver

Dong J, Moayedi RZ, Ong YS, Moosavi-Dezfooli SM. Allies Teach Better Than Enemies: Inverse Adversaries for Robust Knowledge Distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(6):6557-6569. doi:10.1109/tpami.2026.3660863

BibTeX
@article{Dong2026,
  author = {Dong, Junhao and Moayedi, Raoof Zare and Ong, Yew-Soon and Moosavi-Dezfooli, Seyed-Mohsen},
  title = {Allies Teach Better Than Enemies: Inverse Adversaries for Robust Knowledge Distillation},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  volume = {48},
  number = {6},
  pages = {6557--6569},
  doi = {10.1109/tpami.2026.3660863}
}
Kostenlos starten →