Exploiting Intel Advanced Matrix Extensions (AMX) for Large Language Model Inference
IEEE Computer Architecture Letters · 2024-01 · 23 (1) · 117-120
DOI: 10.1109/lca.2024.3397747 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Computer Architecture Letters, 2024-01. Verfasst von Hyungyo Kim und 4 weiteren Autor:innen. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 6 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: IBM IIDAI and Google.
Abstract (Originalsprache)
The ever-increasing number of parameters in Large Language Models (LLMs) demands many expensive GPUs for both inference and training. This is because even such a high-end GPU such as NVIDIA A100 can store only a subset of parameters due to its limited memory capacity. To reduce the number of required GPUs, especially for inference, we may exploit the large memory capacity of (host) CPU to store not only all the model parameters but also intermediate outputs which also require a substantial memory capacity. However, this necessitates frequent data transfers between CPU and GPU over the slow PCIe interface, creating a bottleneck that hinders the accomplishment of both low latency and high throughput in inference. To address such a challenge, we first propose CPU-GPU cooperative computing that exploits the Advanced Matrix Extensions (AMX) capability of the latest Intel CPU, codenamed Sapphire Rapids (SPR). Second, we propose an adaptive model partitioning policy that determines the layers of a given LLM to be run on CPU and GPU, respectively, based on their memory capacity requirement and arithmetic intensity. As CPU executes the layers with large memory capacity but low arithmetic intensity, the amount of data transferred through the PCIe interface is significantly reduced, thereby improving the LLM inference performance. Our evaluation demonstrates that CPU-GPU cooperative computing, based on this policy, delivers 12.1× lower latency and 5.4× higher throughput than GPU-only computing for OPT-30B inference when both CPU-GPU and GPU-only computing store the model in CPU memory.
Förderung
- IBM IIDAI and Google
Zitieren
APA
Kim, H., Ye, G., Wang, N., Yazdanbakhsh, A., & Kim, N. S. (2024). Exploiting Intel Advanced Matrix Extensions (AMX) for Large Language Model Inference. IEEE Computer Architecture Letters, 23(1), 117-120. https://doi.org/10.1109/lca.2024.3397747
Vancouver
Kim H, Ye G, Wang N, Yazdanbakhsh A, Kim NS. Exploiting Intel Advanced Matrix Extensions (AMX) for Large Language Model Inference. IEEE Computer Architecture Letters. 2024;23(1):117-120. doi:10.1109/lca.2024.3397747
BibTeX
@article{Kim2024,
author = {Kim, Hyungyo and Ye, Gaohan and Wang, Nachuan and Yazdanbakhsh, Amir and Kim, Nam Sung},
title = {Exploiting Intel Advanced Matrix Extensions (AMX) for Large Language Model Inference},
journal = {IEEE Computer Architecture Letters},
year = {2024},
volume = {23},
number = {1},
pages = {117--120},
doi = {10.1109/lca.2024.3397747}
}
EUVIMED