EUVIMEDEUVIMED entdecken
Crossref

Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap

Haiyue Ma, Jian Liu, Ronny Krashinsky

Proceedings of the ACM on Measurement and Analysis of Computing Systems · 29.05.2026 · 10 (2) · 1-24

DOI: 10.1145/3805645 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Measurement and Analysis of Computing Systems, online erschienen am 29.05.2026. Verfasst von Haiyue Ma und 2 weiteren Autor:innen. Die Arbeit nennt 42 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

When introducing new operators into Large Language Model (LLM) workloads, a key question is whether fusing the operator into a dependent kernel or overlapping its execution with another independent operator will yield better hardware efficiency. In this work, we analyze these optimization options for the Dropout operator, evaluating fine-grained architecture resource constraints and performance impacts beyond traditional compute or memory metrics. When enabled, Dropout is likely to dramatically impact the performance of end-to-end LLM training. The state-of-the-art optimization is to fuse Dropout into the Flash-Attention kernel; however, fusion fails to fully hide the latency of Random Number Generation (RNG), Dropout's main runtime contributor, due to shared lower-level architecture bottlenecks which are often overlooked. We propose overlapping RNG with preceding independent operators, specifically GEneral Matrix Multiplication (GEMM) layers, to hide RNG latency and improve end-to-end performance. RNG and GEMM have distinct resource requirements and hardware bottlenecks, so they can run together without compromising each other's performance. Our analytical performance model, validated on GH100 GPUs with FP8 precision, shows 1.26x speedup for overlapping RNG and GEMM layers over sequential execution and 1.22x over state-of-the-art fusion approach on Llama3 for a single Transformer block. Our methodology can be generalized to determine when to overlap or fuse new LLM operators across various architectures, models, and configurations.

Zitieren

APA

Ma, H., Liu, J., & Krashinsky, R. (2026). Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 10(2), 1-24. https://doi.org/10.1145/3805645

Vancouver

Ma H, Liu J, Krashinsky R. Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap. Proceedings of the ACM on Measurement and Analysis of Computing Systems. 2026;10(2):1-24. doi:10.1145/3805645

BibTeX
@article{Ma2026,
  author = {Ma, Haiyue and Liu, Jian and Krashinsky, Ronny},
  title = {Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap},
  journal = {Proceedings of the ACM on Measurement and Analysis of Computing Systems},
  year = {2026},
  volume = {10},
  number = {2},
  pages = {1--24},
  doi = {10.1145/3805645}
}
Kostenlos starten →