FedDKD: Federated learning with decentralized knowledge distillation
Applied Intelligence · 2023-08 · 53 (15) · 18547-18563
DOI: 10.1007/s10489-022-04431-1 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Intelligence, online erschienen am 03.02.2023. Verfasst von Xinjia Li und 2 weiteren Autor:innen. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 45 Literaturangaben. Gefördert von: Key Technologies Research and Development Program, Innovative Research Group Project of the National Natural Science Foundation of China, the Shanghai Municipal Science and Technology Major Project.
Abstract (Originalsprache)
The heterogeneity of the data distribution generally influences federated learning performance in neural networks. For a well-performing global model, taking a weighted average of the local models, as in most existing federated learning algorithms, may not guarantee consistency with local models in the space of neural network maps. In this paper, we highlight the significance of the space of neural network maps to relieve the performance decay produced by data heterogeneity and propose a novel federated learning framework equipped with the decentralized knowledge distillation process (FedDKD). In FedDKD, we introduce a decentralized knowledge distillation (DKD) module to distill the knowledge of local models to teach the global model approaching the neural network map average by optimizing the divergence defined in the loss function, other than only averaging parameters as in the literature. Numerical experiments on various heterogeneous datasets reveal that FedDKD outperforms the state-of-the-art methods, especially on some extremely heterogeneous datasets.
Förderung
- Key Technologies Research and Development Program
- Innovative Research Group Project of the National Natural Science Foundation of China
- the Shanghai Municipal Science and Technology Major Project
- the Technology Commission of Shanghai Municipality
Zitieren
APA
Li, X., Chen, B., & Lu, W. (2023). FedDKD: Federated learning with decentralized knowledge distillation. Applied Intelligence, 53(15), 18547-18563. https://doi.org/10.1007/s10489-022-04431-1
Vancouver
Li X, Chen B, Lu W. FedDKD: Federated learning with decentralized knowledge distillation. Applied Intelligence. 2023;53(15):18547-18563. doi:10.1007/s10489-022-04431-1
BibTeX
@article{Li2023,
author = {Li, Xinjia and Chen, Boyu and Lu, Wenlian},
title = {FedDKD: Federated learning with decentralized knowledge distillation},
journal = {Applied Intelligence},
year = {2023},
volume = {53},
number = {15},
pages = {18547--18563},
doi = {10.1007/s10489-022-04431-1}
}
EUVIMED