Server-Client Collaborative Distillation for Federated Reinforcement Learning
ACM Transactions on Knowledge Discovery from Data · 31.01.2024 · 18 (1) · 1-22
DOI: 10.1145/3604939 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Knowledge Discovery from Data, online erschienen am 10.08.2023. Verfasst von Weiming Mai und 5 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 49 Literaturangaben. Gefördert von: NSFC Young Scientists Fund, Guangdong Basic and Applied Basic Research Foundation, RGC Early Career Scheme.
Abstract (Originalsprache)
Federated Learning (FL) learns a global model in a distributional manner, which does not require local clients to share private data. Such merit has drawn lots of attention in the interaction scenarios, where Federated Reinforcement Learning (FRL) emerges as a cross-field research direction focusing on the robust training of agents. Different from FL, the heterogeneity problem in FRL is more challenging because the data depends on the policy of agents and the environment dynamics. FRL learns to interact under the non-stationary environment feedback, while the typical FL methods aim at handling the constant data heterogeneity. In this article, we are among the first attempts to analyze the heterogeneity problem in FRL and propose an off-policy FRL framework. Specifically, a student–teacher–student model learning and fusion method, termed as Server-Client Collaborative Distillation (SCCD), is introduced. Unlike the traditional FL, we distill all local models on the server side for model fusion. To reduce the variance of the training, a local distillation is also conducted every time the agent receives the global model. Experimentally, we compare SCCD with a range of straightforward combinations between FL methods and RL. The results demonstrate that SCCD has a superior performance in four classical continuous control tasks with non-IID environments.
Förderung
- NSFC Young Scientists Fund
- Guangdong Basic and Applied Basic Research Foundation
- RGC Early Career Scheme
- CAAI-Huawei MindSpore Open Fund, and HKBU CSD Departmental Incentive Grant
- NSFC/Research Grants Council (RGC) Joint Research Scheme
- General Research Fund of RGC
- Hong Kong Baptist University
- National Key R&D Program of China
Zitieren
APA
Mai, W., Yao, J., Chen, G., Zhang, Y., Cheung, Y., & Han, B. (2024). Server-Client Collaborative Distillation for Federated Reinforcement Learning. ACM Transactions on Knowledge Discovery from Data, 18(1), 1-22. https://doi.org/10.1145/3604939
Vancouver
Mai W, Yao J, Chen G, Zhang Y, Cheung Y, Han B. Server-Client Collaborative Distillation for Federated Reinforcement Learning. ACM Transactions on Knowledge Discovery from Data. 2024;18(1):1-22. doi:10.1145/3604939
BibTeX
@article{Mai2024,
author = {Mai, Weiming and Yao, Jiangchao and Chen, Gong and Zhang, Ya and Cheung, Yiu-Ming and Han, Bo},
title = {Server-Client Collaborative Distillation for Federated Reinforcement Learning},
journal = {ACM Transactions on Knowledge Discovery from Data},
year = {2024},
volume = {18},
number = {1},
pages = {1--22},
doi = {10.1145/3604939}
}
EUVIMED