EUVIMEDEUVIMED entdecken
Crossref

HFedMoE: Resource-Aware Heterogeneous Federated Learning With Mixture-of-Experts

Zihan Fang, Zheng Lin, Senkang Hu, Yanan Ma, Yihang Tao, Yiqin Deng, Xianhao Chen, Yuguang Fang

IEEE Transactions on Mobile Computing · 2026-12 · 25 (12) · 23221-23236

DOI: 10.1109/tmc.2026.3725213 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Mobile Computing, 2026-12. Verfasst von Zihan Fang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 44 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Research Grants Council of Hong Kong, JC STEM Lab of Smart City funded by The Hong Kong Jockey Club Charities Trust.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

While federated learning (FL) enables fine-tuning of large language models (LLMs) without compromising data privacy, the substantial size of an LLM renders on-device training impractical for resource-constrained clients, such as mobile devices. Thus, Mixture-of-Experts (MoE) models have emerged as a computation-efficient solution, which activates only a sparse subset of experts during model training to reduce computing burden without sacrificing performance. Though integrating MoE into FL fine-tuning holds significant potential, it still encounters three key challenges: i) selecting appropriate experts for clients remains challenging due to the lack of a reliable metric to measure each expert’s impact on local fine-tuning performance, ii) the heterogeneous computing resources across clients severely hinder MoE-based LLM fine-tuning, as dynamic expert activations across diverse input samples can overwhelm resource-constrained devices, and iii) client-specific expert subsets and routing preference undermine global aggregation, where misaligned expert updates and inconsistent gating networks introduce destructive interference. To address these challenges, we propose HFedMoE, a heterogeneous MoE-based FL fine-tuning framework that customizes a subset of experts to each client for computation-efficient LLM fine-tuning. Specifically, HFedMoE identifies the expert importance based on its contributions to fine-tuning performance, and then adaptively selects a subset of experts from an information bottleneck perspective to align with each client’s computing budget. A sparsity-aware model aggregation strategy is also designed to aggregate the actively fine-tuned experts and gating parameters with importance-weighted contributions. Extensive experiments demonstrate that HFedMoE outperforms state-of-the-art benchmarks in training accuracy and convergence speed.

Förderung

  • National Natural Science Foundation of China
  • Research Grants Council of Hong Kong
  • JC STEM Lab of Smart City funded by The Hong Kong Jockey Club Charities Trust
  • Hong Kong SAR Government through Global STEM Professorship and Research Talent Hub
  • Hong Kong Innovation and Technology Commission through InnoHK Project CIMDA

Zitieren

APA

Fang, Z., Lin, Z., Hu, S., Ma, Y., Tao, Y., Deng, Y., Chen, X., & Fang, Y. (2026). HFedMoE: Resource-Aware Heterogeneous Federated Learning With Mixture-of-Experts. IEEE Transactions on Mobile Computing, 25(12), 23221-23236. https://doi.org/10.1109/tmc.2026.3725213

Vancouver

Fang Z, Lin Z, Hu S, Ma Y, Tao Y, Deng Y, et al. HFedMoE: Resource-Aware Heterogeneous Federated Learning With Mixture-of-Experts. IEEE Transactions on Mobile Computing. 2026;25(12):23221-23236. doi:10.1109/tmc.2026.3725213

BibTeX
@article{Fang2026,
  author = {Fang, Zihan and Lin, Zheng and Hu, Senkang and Ma, Yanan and Tao, Yihang and Deng, Yiqin and Chen, Xianhao and Fang, Yuguang},
  title = {HFedMoE: Resource-Aware Heterogeneous Federated Learning With Mixture-of-Experts},
  journal = {IEEE Transactions on Mobile Computing},
  year = {2026},
  volume = {25},
  number = {12},
  pages = {23221--23236},
  doi = {10.1109/tmc.2026.3725213}
}
Kostenlos starten →