FedCD: Towards Consolidated Distillation for Heterogeneous Federated Learning
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (28) · 23256-23264
DOI: 10.1609/aaai.v40i28.39494 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Yichen Li und 7 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).
Abstract (Originalsprache)
Knowledge Distillation (KD) serves as an effective approach to addressing heterogeneity issues in Federated Learning (FL), leveraging additional datasets to align local and global models better. There are two primary distillation paradigms: feature-based distillation, which utilizes intermediate-layer features of the network, and logit-based distillation, which employs the final layer's logit outputs. However, existing studies often select distillation methods based on intuitive and empirical evidence when facing different heterogeneous settings, neglecting the intrinsic relationship between distillation paradigms and heterogeneity. This oversight may result in suboptimal federated knowledge distillation performance under heterogeneous conditions. In this paper, we propose the Consolidated Distillation for Heterogeneous Federated Learning - FedCD that balances knowledge representations from both feature-based and logit-based distillation to enhance performance. Specifically, to address the misalignment between knowledge conveyed by features and logits, we aggregate features from different layers via cross-layer attention to preserve semantic knowledge, followed by distribution modeling using Gaussian Mixture Models. This process strengthens knowledge distillation by constraining the transformation of different network layers' features under a consolidated distribution, thereby mitigating impacts from both data and model heterogeneity. Extensive experiments demonstrate that FedCD outperforms state-of-the-art methods by over 10.72% and validate the effectiveness of our approach.
Zitieren
APA
Li, Y., Su, H., Li, H., Yang, H., Zhuang, X., Xue, H., Wang, H., & Razzak, I. (2026). FedCD: Towards Consolidated Distillation for Heterogeneous Federated Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 40(28), 23256-23264. https://doi.org/10.1609/aaai.v40i28.39494
Vancouver
Li Y, Su H, Li H, Yang H, Zhuang X, Xue H, et al. FedCD: Towards Consolidated Distillation for Heterogeneous Federated Learning. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(28):23256-23264. doi:10.1609/aaai.v40i28.39494
BibTeX
@article{Li2026,
author = {Li, Yichen and Su, Hang and Li, Huifa and Yang, Haolin and Zhuang, Xinlin and Xue, Haochen and Wang, Haozhao and Razzak, Imran},
title = {FedCD: Towards Consolidated Distillation for Heterogeneous Federated Learning},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {28},
pages = {23256--23264},
doi = {10.1609/aaai.v40i28.39494}
}
EUVIMED