High-Performance Tensor Learning Primitives Using GPU Tensor Cores
IEEE Transactions on Computers · 01.06.2023 · 72 (6) · 1733-1746
DOI: 10.1109/tc.2022.3222955 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Computers, erschienen am 01.06.2023. Verfasst von Xiao-Yang Liu und 5 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 52 Literaturangaben.
Abstract (Originalsprache)
Tensor learning is a powerful tool for big data analytics and machine learning, e.g., gene analysis and deep learning. However, tensor learning algorithms are compute-intensive since their time and space complexities grow exponentially with the order of tensors, which hinders their application. In this paper, we exploit the parallelism of tensor learning primitives using GPU tensor cores and develop high-performance tensor learning algorithms. First, we propose novel hardware-oriented optimization strategies for tensor learning primitives on GPU tensor cores. Second, for big data analytics, we employ the optimized tensor learning primitives to accelerate the CP tensor decomposition and then apply it for gene analysis. Third, we optimize the Tucker tensor decomposition and propose a novel Tucker tensor layer to compress deep neural networks. We employ natural gradients to train the neural networks, which only involve a forward pass without backpropagation and thus are suitable for GPU computations. Compared with TensorLab and TensorLy libraries on an A100 GPU, our third-order CP tensor decomposition achieves up to $16.32\times$ 16 . 32 × and $32.25\times$ 32 . 25 × speedups; and $6.09\times$ 6 . 09 × and $6.72\times$ 6 . 72 × speedups for our third-order Tucker tensor decomposition. The proposed fourth-order CP and Tucker tensor decompositions achieve up to $30.65\times$ 30 . 65 × and $5.41\times$ 5 . 41 × speedups over the TensorLab. Our CP tensor decomposition for gene analysis achieves up to $5.88\times$ 5 . 88 × speedup over TensorLy. Compared with a conventional fully connected neural network, our Tucker tensor layer neural network achieves an accuracy of $97.9\%$ 97 . 9 % , a speedup of $4.47\times$ 4 . 47 × , and a compression ratio of $2.92$ 2 . 92 at the cost of $0.4\%$ 0 . 4 % drop in accuracy.
Zitieren
APA
Liu, X., Zhang, Z., Wang, Z., Lu, H., Wang, X., & Walid, A. (2023). High-Performance Tensor Learning Primitives Using GPU Tensor Cores. IEEE Transactions on Computers, 72(6), 1733-1746. https://doi.org/10.1109/tc.2022.3222955
Vancouver
Liu X, Zhang Z, Wang Z, Lu H, Wang X, Walid A. High-Performance Tensor Learning Primitives Using GPU Tensor Cores. IEEE Transactions on Computers. 2023;72(6):1733-1746. doi:10.1109/tc.2022.3222955
BibTeX
@article{Liu2023,
author = {Liu, Xiao-Yang and Zhang, Zeliang and Wang, Zhiyuan and Lu, Han and Wang, Xiaodong and Walid, Anwar},
title = {High-Performance Tensor Learning Primitives Using GPU Tensor Cores},
journal = {IEEE Transactions on Computers},
year = {2023},
volume = {72},
number = {6},
pages = {1733--1746},
doi = {10.1109/tc.2022.3222955}
}
EUVIMED