PyAO: PyTorch-Based Memory-Efficient LLM Training on Ethernet-Interconnected Clusters
Sensors · 07.04.2026 · 26 (7) · 2269
DOI: 10.3390/s26072269 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 07.04.2026. Verfasst von Daemin Kim und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Engineering, Keimyung University, Daegu 42601, Republic of Korea. Die Arbeit nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Daegu Digital Innovation Promotion Agency(DIP), Daegu Digital Innovation Promotion Agency (DIP) OF FUNDER.
Abstract (Originalsprache)
As large language models (LLMs) pursue higher accuracy, their model sizes have surged, substantially increasing GPU memory consumption. Prior work mitigates this issue by distributing the memory burden across multiple GPUs. However, on clusters interconnected via Ethernet, the resulting computational intensity is insufficient to hide the significant network latency. Achieving a favorable compute-to-communication ratio is further constrained by the memory required to cache the massive activations generated during the forward pass. PyAO, proposed in this paper, effectively offloads activations, selects offloading strategies based on their offloading efficiency, and minimizes data-movement bottlenecks, thereby enabling larger micro-batch sizes. In Ethernet-interconnected cluster environments, experiments on popular models—including OPT-1.3B, GPT-0.8B, and Llama-1.2B—demonstrate that PyAO reduces peak GPU memory by up to 1.94× at the same micro-batch size, enables up to 2.5× larger batch sizes, and accelerates training by up to 3.63× relative to the baseline.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Engineering, Keimyung University, Daegu 42601, Republic of Korea.
ORCID-Profile: Sejin Park
Förderung
- Daegu Digital Innovation Promotion Agency(DIP)
- Daegu Digital Innovation Promotion Agency (DIP) OF FUNDER
Zitieren
APA
Kim, D., Kim, H., Ahn, J., & Park, S. (2026). PyAO: PyTorch-Based Memory-Efficient LLM Training on Ethernet-Interconnected Clusters. Sensors, 26(7), 2269. https://doi.org/10.3390/s26072269
Vancouver
Kim D, Kim H, Ahn J, Park S. PyAO: PyTorch-Based Memory-Efficient LLM Training on Ethernet-Interconnected Clusters. Sensors. 2026;26(7):2269. doi:10.3390/s26072269
BibTeX
@article{Kim2026,
author = {Kim, Daemin and Kim, Hyorim and Ahn, Juncheol and Park, Sejin},
title = {PyAO: PyTorch-Based Memory-Efficient LLM Training on Ethernet-Interconnected Clusters},
journal = {Sensors},
year = {2026},
volume = {26},
number = {7},
pages = {2269},
doi = {10.3390/s26072269}
}
EUVIMED