MoonPoly : Bridging Code Generation and Adaptive Execution via Micro-Kernel Polymerization for Optimizing Dynamic-Shape Tensor Operators
ACM Transactions on Architecture and Code Optimization · 30.06.2026 · 23 (2) · 1-25
DOI: 10.1145/3818618 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Architecture and Code Optimization, online erschienen am 26.06.2026. Verfasst von Yangyu Zhang und 9 weiteren Autor:innen. Die Arbeit nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, China Postdoctoral Science Foundation.
Abstract (Originalsprache)
The prevalence of dynamic tensor shapes, driven by applications like language model serving with varying sequence lengths, is a defining characteristic of modern deep neural networks. This dynamism poses a fundamental challenge: reconciling the need for intensive, offline code generation to achieve peak performance with the demand for low-latency, adaptive execution to handle unpredictable runtime tensor shapes. Consequently, mainstream strategies are ineffective. Vendor-provided libraries, while highly optimized for a subset of common shapes, suffer performance degradation on unconventional ones. Static tensor compilers are hamstrung by prohibitive just-in-time compilation overheads for each new shape. While recent dynamic-shape compilers offer an alternative, they rely on predefined shape ranges, making them brittle when inputs fall outside these bounds. To resolve this tension, we present MoonPoly , a dynamic-shape tensor compiler that introduces micro-kernel polymerization . Our approach decouples these conflicting requirements through a two-stage process. In the offline stage, it performs intensive auto-tuning to generate a set of micro-kernels and corresponding performance models. The online stage then performs adaptive execution, rapidly assembling a near-optimal tensor operator on-the-fly, guided by a lightweight cost model. Evaluated on an NVIDIA A100 GPU, MoonPoly achieves an average operator-level speedup of 1.27× over the cuBLAS library across a diverse set of operators and data types, which in turn yields end-to-end inference acceleration for a variety of models, including BERT, the Vision Transformer, and large language models.
Förderung
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- China Postdoctoral Science Foundation
- SKLP, ICT, CAS
- Australian Research Council
Zitieren
APA
Zhang, Y., Li, G., Yu, F., Luo, F., Sun, Q., Wang, X., Zhao, J., Cui, H., Feng, X., & Xue, J. (2026). MoonPoly : Bridging Code Generation and Adaptive Execution via Micro-Kernel Polymerization for Optimizing Dynamic-Shape Tensor Operators. ACM Transactions on Architecture and Code Optimization, 23(2), 1-25. https://doi.org/10.1145/3818618
Vancouver
Zhang Y, Li G, Yu F, Luo F, Sun Q, Wang X, et al. MoonPoly : Bridging Code Generation and Adaptive Execution via Micro-Kernel Polymerization for Optimizing Dynamic-Shape Tensor Operators. ACM Transactions on Architecture and Code Optimization. 2026;23(2):1-25. doi:10.1145/3818618
BibTeX
@article{Zhang2026,
author = {Zhang, Yangyu and Li, Guangli and Yu, Feng and Luo, Fan and Sun, Qianqi and Wang, Xueying and Zhao, Jiacheng and Cui, Huimin and Feng, Xiaobing and Xue, Jingling},
title = {MoonPoly : Bridging Code Generation and Adaptive Execution via Micro-Kernel Polymerization for Optimizing Dynamic-Shape Tensor Operators},
journal = {ACM Transactions on Architecture and Code Optimization},
year = {2026},
volume = {23},
number = {2},
pages = {1--25},
doi = {10.1145/3818618}
}
EUVIMED