EUVIMEDEUVIMED entdecken
Crossref

Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting

Chunlin Tian, Li Li, Kahou Tam, Yebo Wu, Cheng-Zhong Xu

IEEE Transactions on Parallel and Distributed Systems · 2024-12 · 35 (12) · 2513-2526

DOI: 10.1109/tpds.2024.3480115 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Parallel and Distributed Systems, 2024-12. Verfasst von Chunlin Tian und 4 weiteren Autor:innen. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 75 Literaturangaben. Gefördert von: Start-up Research Grant of University of Macau.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Federated Learning (FL) enables multiple devices to collaboratively train a shared model while preserving data privacy. Ever-increasing model complexity coupled with limited memory resources on the participating devices severely bottlenecks the deployment of FL in real-world scenarios. Thus, a framework that can effectively break the memory wall while jointly taking into account the hardware and statistical heterogeneity in FL is urgently required. In this article, we propose SmartSplit a framework that effectively reduces the memory footprint on the device side while guaranteeing the training progress and model accuracy for heterogeneous FL through model splitting. Towards this end, SmartSplit employs a hierarchical structure to adaptively guide the overall training process. In each training round, the central manager, hosted on the server, dynamically selects the participating devices and sets the cutting layer by jointly considering the memory budget, training capacity, and data distribution of each device. The MEC manager, deployed within the edge server, proceeds to split the local model and perform training of the server-side portion. Meanwhile, it fine-tunes the splitting points based on the time-evolving statistical importance. The on-device manager, embedded inside each mobile device, continuously monitors the local training status while employing cost-aware checkpointing to match the runtime dynamic memory budget. Extensive experiments on representative datasets are conducted on both commercial off-the-shelf mobile device testbeds. The experimental results show that SmartSplit excels in FL training on highly memory-constrained mobile SoCs, offering up to a 94% peak latency reduction and 100-fold memory savings. It enhances accuracy performance by 1.49%-57.18% and adaptively adjusts to dynamic memory budgets through cost-aware recomputation

Förderung

  • Start-up Research Grant of University of Macau

Zitieren

APA

Tian, C., Li, L., Tam, K., Wu, Y., & Xu, C. (2024). Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting. IEEE Transactions on Parallel and Distributed Systems, 35(12), 2513-2526. https://doi.org/10.1109/tpds.2024.3480115

Vancouver

Tian C, Li L, Tam K, Wu Y, Xu C. Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting. IEEE Transactions on Parallel and Distributed Systems. 2024;35(12):2513-2526. doi:10.1109/tpds.2024.3480115

BibTeX
@article{Tian2024,
  author = {Tian, Chunlin and Li, Li and Tam, Kahou and Wu, Yebo and Xu, Cheng-Zhong},
  title = {Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting},
  journal = {IEEE Transactions on Parallel and Distributed Systems},
  year = {2024},
  volume = {35},
  number = {12},
  pages = {2513--2526},
  doi = {10.1109/tpds.2024.3480115}
}
Kostenlos starten →