Heterogeneous Federated Learning Driven by Multi-Knowledge Distillation
IEEE Transactions on Mobile Computing · 2025-12 · 24 (12) · 13048-13061
DOI: 10.1109/tmc.2025.3586921 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Mobile Computing, 2025-12. Verfasst von Bin Xu und 6 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 38 Literaturangaben. Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
In a fully heterogeneous federated learning environment, the client has significant differences in model structure and local data distribution (Non-IID), and the joint learning of the client model is blocked due to the limited communication content available for interaction in a fully heterogeneous scenario. In this context, the global knowledge constructed by the server through the simple aggregation of the client logits is essentially a fuzzy representation containing a lot of noise and information loss, which is difficult to effectively guide the client model update. To solve these problems, this paper proposes a heterogeneous federated learning framework (FedMkd) based on multi-knowledge distillation fusion to cope with multiple challenges in heterogeneous environments. The FedMkd framework uses a class-grained logits interaction architecture (CLIA) and introduces an efficient knowledge sharing mechanism. It innovatively integrates two knowledge distillation methods: 1) Temperature-Adaptive Knowledge Distillation (TAKD), which provides differentiated temperatures for teacher and student models by adaptively adjusting the distillation temperature, maximizing knowledge transfer between them; 2) Class-related Knowledge Distillation (CRKD), which introduces batch-level sample correlation loss to reduce over-reliance on specific samples or classes and improve the model’s understanding of overall data features. We conducted a large number of experiments on four public datasets. The results show that in a variety of data and model heterogeneous scenarios, FedMkd still performs better than the comparison method when the communication overhead is reduced by more than one order of magnitude.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Xu, B., Cheng, L., Wen, Q., Zou, Z., Hu, X., Dong, Z., & Qi, J. (2025). Heterogeneous Federated Learning Driven by Multi-Knowledge Distillation. IEEE Transactions on Mobile Computing, 24(12), 13048-13061. https://doi.org/10.1109/tmc.2025.3586921
Vancouver
Xu B, Cheng L, Wen Q, Zou Z, Hu X, Dong Z, et al. Heterogeneous Federated Learning Driven by Multi-Knowledge Distillation. IEEE Transactions on Mobile Computing. 2025;24(12):13048-13061. doi:10.1109/tmc.2025.3586921
BibTeX
@article{Xu2025,
author = {Xu, Bin and Cheng, Longgang and Wen, Qing and Zou, Zhensheng and Hu, Xiaoxuan and Dong, Zhenjiang and Qi, Jin},
title = {Heterogeneous Federated Learning Driven by Multi-Knowledge Distillation},
journal = {IEEE Transactions on Mobile Computing},
year = {2025},
volume = {24},
number = {12},
pages = {13048--13061},
doi = {10.1109/tmc.2025.3586921}
}
EUVIMED