Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap
Proceedings of the ACM on Measurement and Analysis of Computing Systems · 29.05.2026 · 10 (2) · 1-24
DOI: 10.1145/3805645 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Measurement and Analysis of Computing Systems, online erschienen am 29.05.2026. Verfasst von Haiyue Ma und 2 weiteren Autor:innen. Die Arbeit nennt 42 Literaturangaben.
Abstract (Originalsprache)
When introducing new operators into Large Language Model (LLM) workloads, a key question is whether fusing the operator into a dependent kernel or overlapping its execution with another independent operator will yield better hardware efficiency. In this work, we analyze these optimization options for the Dropout operator, evaluating fine-grained architecture resource constraints and performance impacts beyond traditional compute or memory metrics. When enabled, Dropout is likely to dramatically impact the performance of end-to-end LLM training. The state-of-the-art optimization is to fuse Dropout into the Flash-Attention kernel; however, fusion fails to fully hide the latency of Random Number Generation (RNG), Dropout's main runtime contributor, due to shared lower-level architecture bottlenecks which are often overlooked. We propose overlapping RNG with preceding independent operators, specifically GEneral Matrix Multiplication (GEMM) layers, to hide RNG latency and improve end-to-end performance. RNG and GEMM have distinct resource requirements and hardware bottlenecks, so they can run together without compromising each other's performance. Our analytical performance model, validated on GH100 GPUs with FP8 precision, shows 1.26x speedup for overlapping RNG and GEMM layers over sequential execution and 1.22x over state-of-the-art fusion approach on Llama3 for a single Transformer block. Our methodology can be generalized to determine when to overlap or fuse new LLM operators across various architectures, models, and configurations.
Zitieren
APA
Ma, H., Liu, J., & Krashinsky, R. (2026). Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 10(2), 1-24. https://doi.org/10.1145/3805645
Vancouver
Ma H, Liu J, Krashinsky R. Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap. Proceedings of the ACM on Measurement and Analysis of Computing Systems. 2026;10(2):1-24. doi:10.1145/3805645
BibTeX
@article{Ma2026,
author = {Ma, Haiyue and Liu, Jian and Krashinsky, Ronny},
title = {Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap},
journal = {Proceedings of the ACM on Measurement and Analysis of Computing Systems},
year = {2026},
volume = {10},
number = {2},
pages = {1--24},
doi = {10.1145/3805645}
}
EUVIMED