Defending against knowledge distillation-based model extraction via feature covariance disalignment
The Computer Journal · 15.11.2026 · 69 (11) · 1855-1864
DOI: 10.1093/comjnl/bxag063 ↗Kurz gefasst
Fachartikel in der Zeitschrift The Computer Journal, online erschienen am 02.08.2026. Verfasst von Yao Zhang und 2 weiteren Autor:innen. Die Arbeit nennt 36 Literaturangaben.
Abstract (Originalsprache)
Current research on knowledge distillation, especially data-free knowledge distillation, is getting deeper and deeper, which may lead to an increase in model stealing. To address this challenge, anti-distillation techniques have been developed. Anti-distillation aims to eliminate dark knowledge in teacher models in order to prevent potential thieves from illegally accessing the feature extraction capabilities of the models via knowledge distillation. Traditional anti-distillation methods apply perturbations in the output, but this indirectly affects the model feature layer and fails to completely remove the dark knowledge in the feature layer, resulting in limited effectiveness when counteracting feature distillation methods. To solve this problem, this paper innovatively introduces perturbations in the feature layer, utilizes the covariance matrix to describe the global relationship of the feature map, and extracts the subspace using eigenvalue to improve the training efficiency. Eventually, the dark knowledge in the anti-distillation model is effectively eliminated by the inverse alignment strategy in both scale and angle dimensions. Experimental results show that our method performs well on multiple datasets, such as CIFAR-100 and Tiny-ImageNet, and can effectively resist different state-of-the-art distillation methods including logits distillation, feature distillation, and data-free distillation, and achieves stronger anti-distillation capability and model performance.
Zitieren
APA
Zhang, Y., Li, Y., & Pan, Z. (2026). Defending against knowledge distillation-based model extraction via feature covariance disalignment. The Computer Journal, 69(11), 1855-1864. https://doi.org/10.1093/comjnl/bxag063
Vancouver
Zhang Y, Li Y, Pan Z. Defending against knowledge distillation-based model extraction via feature covariance disalignment. The Computer Journal. 2026;69(11):1855-1864. doi:10.1093/comjnl/bxag063
BibTeX
@article{Zhang2026,
author = {Zhang, Yao and Li, Yang and Pan, Zhisong},
title = {Defending against knowledge distillation-based model extraction via feature covariance disalignment},
journal = {The Computer Journal},
year = {2026},
volume = {69},
number = {11},
pages = {1855--1864},
doi = {10.1093/comjnl/bxag063}
}
EUVIMED