EUVIMEDEUVIMED entdecken
Crossref

MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training

Pinxue Zhao, Hailin Zhang, Fangcheng Fu, Xiaonan Nie, Qibin Liu, Fang Yang, Yuanbo Peng, Dian Jiao, Shuaipeng Li, Jinbao Xue, Yangyu Tao, Bin Cui

Proceedings of the ACM on Management of Data · 10.02.2025 · 3 (1) · 1-28

DOI: 10.1145/3709703 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Management of Data, online erschienen am 11.02.2025. Verfasst von Pinxue Zhao und 11 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 104 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Nowadays, Large Language Models (LLMs) have been trained using extended context lengths to foster more creative applications. However, long context training poses great challenges considering the constraint of GPU memory. It not only leads to substantial activation memory consumption during training, but also incurs considerable memory fragmentation. To facilitate long context training, existing frameworks have adopted strategies such as recomputation and various forms of parallelisms. Nevertheless, these techniques rely on redundant computation or extensive communication, resulting in low Model FLOPS Utilization (MFU). In this paper, we propose MEMO, a novel LLM training framework designed for fine-grained activation memory management. Given the quadratic scaling of computation and linear scaling of memory with sequence lengths when using FlashAttention, we offload memory-consuming activations to CPU memory after each layer's forward pass and fetch them during the backward pass. To maximize the swapping of activations without hindering computation, and to avoid exhausting limited CPU memory, we implement a token-wise activation recomputation and swapping mechanism. Furthermore, we tackle the memory fragmentation issue by employing a bi-level Mixed Integer Programming (MIP) approach, optimizing memory reuse across transformer layers. Empirical results demonstrate that MEMO achieves an average of 1.97x and 1.80x MFU compared to Megatron-LM and DeepSpeed, respectively. This improvement is attributed to MEMO's ability to minimize memory fragmentation, reduce recomputation and intensive communication, and circumvent the delays associated with the memory reorganization process due to fragmentation. By leveraging fine-grained activation memory management, MEMO facilitates efficient training of 7B LLM with 1 million sequence length on just 8 A800 GPUs, achieving an MFU of 52.30%.

Zitieren

APA

Zhao, P., Zhang, H., Fu, F., Nie, X., Liu, Q., Yang, F., Peng, Y., Jiao, D., Li, S., Xue, J., Tao, Y., & Cui, B. (2025). MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training. Proceedings of the ACM on Management of Data, 3(1), 1-28. https://doi.org/10.1145/3709703

Vancouver

Zhao P, Zhang H, Fu F, Nie X, Liu Q, Yang F, et al. MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training. Proceedings of the ACM on Management of Data. 2025;3(1):1-28. doi:10.1145/3709703

BibTeX
@article{Zhao2025,
  author = {Zhao, Pinxue and Zhang, Hailin and Fu, Fangcheng and Nie, Xiaonan and Liu, Qibin and Yang, Fang and Peng, Yuanbo and Jiao, Dian and Li, Shuaipeng and Xue, Jinbao and Tao, Yangyu and Cui, Bin},
  title = {MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training},
  journal = {Proceedings of the ACM on Management of Data},
  year = {2025},
  volume = {3},
  number = {1},
  pages = {1--28},
  doi = {10.1145/3709703}
}
Kostenlos starten →