EUVIMEDEUVIMED entdecken
Crossref

Waterwave: A GPU Memory Flow Engine for Concurrent DNN Training

Xuanhua Shi, Xuan Peng, Ligang He, Yunfei Zhao, Hai Jin

IEEE Transactions on Computers · 2023-10 · 72 (10) · 2938-2950

DOI: 10.1109/tc.2023.3278530 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Computers, 2023-10. Verfasst von Xuanhua Shi und 4 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 42 Literaturangaben. Gefördert von: National Key R&D Program of China, Key R&D Program of Hubei.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Training Deep Neural Networks (DNN) concurrently is becoming increasingly important for deep learning practitioners, e.g., hyperparameter optimization (HPO) and neural architecture search (NAS) . The GPU memory capacity is the impediment that prohibits multiple DNNs from being trained on the same GPU due to the large memory usage during training. In this paper, we propose Waterwave , a GPU memory flow engine for concurrent deep learning training. First, to address the memory explosion brought by the long time lag between memory allocation and deallocation time, we develop an allocator tailored for multi-streams. By making the allocator aware of the stream information, a prioritized allocation is conducted based on the chunk's synchronization attributes, allowing us to provide useable memory after scheduling rather than waiting it to be really released after GPU computation. Second, Waterwave partitions the compute graph to a set of continuous node groups and then performs finer-grained scheduling: NodeGroup pipeline execution , to guarantee a proper memory requests order. Waterwave can accomplish up to 96.8% of the maximum batch size of solo training. Additionally, in scenarios with high memory demand, Waterwave can outperform existing spatial sharing and temporal sharing by up to 12x and 1.49x, respectively.

Förderung

  • National Key R&D Program of China
  • Key R&D Program of Hubei

Zitieren

APA

Shi, X., Peng, X., He, L., Zhao, Y., & Jin, H. (2023). Waterwave: A GPU Memory Flow Engine for Concurrent DNN Training. IEEE Transactions on Computers, 72(10), 2938-2950. https://doi.org/10.1109/tc.2023.3278530

Vancouver

Shi X, Peng X, He L, Zhao Y, Jin H. Waterwave: A GPU Memory Flow Engine for Concurrent DNN Training. IEEE Transactions on Computers. 2023;72(10):2938-2950. doi:10.1109/tc.2023.3278530

BibTeX
@article{Shi2023,
  author = {Shi, Xuanhua and Peng, Xuan and He, Ligang and Zhao, Yunfei and Jin, Hai},
  title = {Waterwave: A GPU Memory Flow Engine for Concurrent DNN Training},
  journal = {IEEE Transactions on Computers},
  year = {2023},
  volume = {72},
  number = {10},
  pages = {2938--2950},
  doi = {10.1109/tc.2023.3278530}
}
Kostenlos starten →