EUVIMEDEUVIMED entdecken
Crossref

Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation

Zhiyuan Wu, Sheng Sun, Yuwei Wang, Min Liu, Quyang Pan, Junbo Zhang, Zeju Li, Qingxiang Liu

ACM Transactions on Intelligent Systems and Technology · 30.04.2024 · 15 (2) · 1-34

DOI: 10.1145/3639369 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Intelligent Systems and Technology, online erschienen am 22.02.2024. Verfasst von Zhiyuan Wu und 7 weiteren Autor:innen. Die Arbeit wurde bisher 27-mal zitiert (Crossref) und nennt 39 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, Innovation Capability Support Program of Shaanxi.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Federated learning (FL) is a privacy-preserving machine learning paradigm in which the server periodically aggregates local model parameters from cli ents without assembling their private data. Constrained communication and personalization requirements pose severe challenges to FL. Federated distillation (FD) is proposed to simultaneously address the above two problems, which exchanges knowledge between the server and clients, supporting heterogeneous local models while significantly reducing communication overhead. However, most existing FD methods require a proxy dataset, which is often unavailable in reality. A few recent proxy-data-free FD approaches can eliminate the need for additional public data, but suffer from remarkable discrepancy among local knowledge due to client-side model heterogeneity, leading to ambiguous representation on the server and inevitable accuracy degradation. To tackle this issue, we propose a proxy-data-free FD algorithm based on distributed knowledge congruence (FedDKC). FedDKC leverages well-designed refinement strategies to narrow local knowledge differences into an acceptable upper bound, so as to mitigate the negative effects of knowledge incongruence. Specifically, from perspectives of peak probability and Shannon entropy of local knowledge, we design kernel-based knowledge refinement (KKR) and searching-based knowledge refinement (SKR) respectively, and theoretically guarantee that the refined-local knowledge can satisfy an approximately-similar distribution and be regarded as congruent. Extensive experiments conducted on three common datasets demonstrate that our proposed FedDKC significantly outperforms the state-of-the-art on various heterogeneous settings while evidently improving the convergence speed.

Förderung

  • National Key Research and Development Program of China
  • National Natural Science Foundation of China
  • Innovation Capability Support Program of Shaanxi
  • Shaanxi Qinchuangyuan “scientists+engineers” team
  • Innovation Funding of ICT, CAS
  • Beijing Natural Science Foundation
  • Beijing Science and Technology Project

Zitieren

APA

Wu, Z., Sun, S., Wang, Y., Liu, M., Pan, Q., Zhang, J., Li, Z., & Liu, Q. (2024). Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation. ACM Transactions on Intelligent Systems and Technology, 15(2), 1-34. https://doi.org/10.1145/3639369

Vancouver

Wu Z, Sun S, Wang Y, Liu M, Pan Q, Zhang J, et al. Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation. ACM Transactions on Intelligent Systems and Technology. 2024;15(2):1-34. doi:10.1145/3639369

BibTeX
@article{Wu2024,
  author = {Wu, Zhiyuan and Sun, Sheng and Wang, Yuwei and Liu, Min and Pan, Quyang and Zhang, Junbo and Li, Zeju and Liu, Qingxiang},
  title = {Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation},
  journal = {ACM Transactions on Intelligent Systems and Technology},
  year = {2024},
  volume = {15},
  number = {2},
  pages = {1--34},
  doi = {10.1145/3639369}
}
Kostenlos starten →