EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap

Haiyue Ma, Jian Liu, Ronny Krashinsky

ACM SIGMETRICS Performance Evaluation Review · 20.07.2026 · 54 (1) · 235-237

DOI: 10.1145/3820171.3806911 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM SIGMETRICS Performance Evaluation Review, online erschienen am 27.07.2026. Verfasst von Haiyue Ma und 2 weiteren Autor:innen. Die Arbeit nennt 13 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This work proposes overlapping Random Number Generation (RNG), the main runtime contributor of Dropout, with preceding GEneral Matrix Multiplication (GEMM) layers to hide RNG latency during LLM training. The state-of-the-art optimization is to fuse Dropout into the Flash-Attention kernel; however, evaluating fine-grained architecture resource constraints beyond traditional compute or memory metrics reveals that fusion fails to fully hide RNG latency due to shared lower-level architecture bottlenecks. RNG and GEMM have distinct hardware bottlenecks, so they can run together without compromising each other's performance. Our analytical model, validated on GH100 GPUs, shows 1.26× speedup over sequential execution and 1.22× over state-of-the-art fusion on Llama3 for a single Transformer block. Our methodology generalizes to fusion-versus-overlap decisions for new LLM operators across various architectures, models, and configurations.

Zitieren

APA

Ma, H., Liu, J., & Krashinsky, R. (2026). Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap. ACM SIGMETRICS Performance Evaluation Review, 54(1), 235-237. https://doi.org/10.1145/3820171.3806911

Vancouver

Ma H, Liu J, Krashinsky R. Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap. ACM SIGMETRICS Performance Evaluation Review. 2026;54(1):235-237. doi:10.1145/3820171.3806911

BibTeX
@article{Ma2026,
  author = {Ma, Haiyue and Liu, Jian and Krashinsky, Ronny},
  title = {Optimizing Dropout in LLM Training: Performance Comparison of Fusion and Overlap},
  journal = {ACM SIGMETRICS Performance Evaluation Review},
  year = {2026},
  volume = {54},
  number = {1},
  pages = {235--237},
  doi = {10.1145/3820171.3806911}
}
Kostenlos starten →