Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap
ACM SIGMETRICS Performance Evaluation Review · 20.07.2026 · 54 (1) · 235-237
DOI: 10.1145/3820171.3806911 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM SIGMETRICS Performance Evaluation Review, online erschienen am 27.07.2026. Verfasst von Haiyue Ma und 2 weiteren Autor:innen. Die Arbeit nennt 13 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
This work proposes overlapping Random Number Generation (RNG), the main runtime contributor of Dropout, with preceding GEneral Matrix Multiplication (GEMM) layers to hide RNG latency during LLM training. The state-of-the-art optimization is to fuse Dropout into the Flash-Attention kernel; however, evaluating fine-grained architecture resource constraints beyond traditional compute or memory metrics reveals that fusion fails to fully hide RNG latency due to shared lower-level architecture bottlenecks. RNG and GEMM have distinct hardware bottlenecks, so they can run together without compromising each other's performance. Our analytical model, validated on GH100 GPUs, shows 1.26× speedup over sequential execution and 1.22× over state-of-the-art fusion on Llama3 for a single Transformer block. Our methodology generalizes to fusion-versus-overlap decisions for new LLM operators across various architectures, models, and configurations.
Zitieren
APA
Ma, H., Liu, J., & Krashinsky, R. (2026). Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap. ACM SIGMETRICS Performance Evaluation Review, 54(1), 235-237. https://doi.org/10.1145/3820171.3806911
Vancouver
Ma H, Liu J, Krashinsky R. Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap. ACM SIGMETRICS Performance Evaluation Review. 2026;54(1):235-237. doi:10.1145/3820171.3806911
BibTeX
@article{Ma2026,
author = {Ma, Haiyue and Liu, Jian and Krashinsky, Ronny},
title = {Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap},
journal = {ACM SIGMETRICS Performance Evaluation Review},
year = {2026},
volume = {54},
number = {1},
pages = {235--237},
doi = {10.1145/3820171.3806911}
}
EUVIMED