ParaDySe: A Parallel Strategy Switching Framework for Dynamic Sequences in Transformer-based Large Language Models
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (29) · 24648-24655
DOI: 10.1609/aaai.v40i29.39649 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Zhixin Ou und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).
Abstract (Originalsprache)
Dynamic sequences with varying lengths have been widely used in the training of Transformer-based large language models (LLMs). However, current training frameworks adopt a pre-defined static parallel strategy for these sequences, causing neither communication-parallelization cancellation on short sequences nor out-of-memory on long sequences. To mitigate these issues, we propose ParaDySe, a novel adaptive Parallel strategy switching framework for Dynamic Sequences. ParaDySe enables on-the-fly optimal strategy adoption according to the immediate input sequence. It first implements the modular function libraries for parallel strategies with unified tensor layout specifications, and then builds sequence-aware memory and time cost models with hybrid methods. Guided by cost models, ParaDySe selects optimal layer-wise strategies for dynamic sequences via an efficient heuristic algorithm. By integrating these techniques together, ParaDySe achieves seamless hot-switching of optimal strategies through its well-designed function libraries. We compare ParaDySe with baselines on representative LLMs under datasets with sequence lengths up to 624K. Experimental results indicate that ParaDySe addresses OOM and CPC bottlenecks in LLM training by systematically integrating long-sequence optimizations with existing frameworks.
Zitieren
APA
Ou, Z., Liang, P., Qiao, L., Han, J., & Liu, B. (2026). ParaDySe: A Parallel Strategy Switching Framework for Dynamic Sequences in Transformer-based Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 40(29), 24648-24655. https://doi.org/10.1609/aaai.v40i29.39649
Vancouver
Ou Z, Liang P, Qiao L, Han J, Liu B. ParaDySe: A Parallel Strategy Switching Framework for Dynamic Sequences in Transformer-based Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(29):24648-24655. doi:10.1609/aaai.v40i29.39649
BibTeX
@article{Ou2026,
author = {Ou, Zhixin and Liang, Peng and Qiao, Linbo and Han, Jianchen and Liu, Baihui},
title = {ParaDySe: A Parallel Strategy Switching Framework for Dynamic Sequences in Transformer-based Large Language Models},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {29},
pages = {24648--24655},
doi = {10.1609/aaai.v40i29.39649}
}
EUVIMED