EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Exploiting Intel Advanced Matrix Extensions (AMX) for Large Language Model Inference

Hyungyo Kim, Gaohan Ye, Nachuan Wang, Amir Yazdanbakhsh, Nam Sung Kim

IEEE Computer Architecture Letters · 2024-01 · 23 (1) · 117-120

DOI: 10.1109/lca.2024.3397747 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Computer Architecture Letters, 2024-01. Verfasst von Hyungyo Kim und 4 weiteren Autor:innen. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 6 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: IBM IIDAI and Google.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The ever-increasing number of parameters in Large Language Models (LLMs) demands many expensive GPUs for both inference and training. This is because even such a high-end GPU such as NVIDIA A100 can store only a subset of parameters due to its limited memory capacity. To reduce the number of required GPUs, especially for inference, we may exploit the large memory capacity of (host) CPU to store not only all the model parameters but also intermediate outputs which also require a substantial memory capacity. However, this necessitates frequent data transfers between CPU and GPU over the slow PCIe interface, creating a bottleneck that hinders the accomplishment of both low latency and high throughput in inference. To address such a challenge, we first propose CPU-GPU cooperative computing that exploits the Advanced Matrix Extensions (AMX) capability of the latest Intel CPU, codenamed Sapphire Rapids (SPR). Second, we propose an adaptive model partitioning policy that determines the layers of a given LLM to be run on CPU and GPU, respectively, based on their memory capacity requirement and arithmetic intensity. As CPU executes the layers with large memory capacity but low arithmetic intensity, the amount of data transferred through the PCIe interface is significantly reduced, thereby improving the LLM inference performance. Our evaluation demonstrates that CPU-GPU cooperative computing, based on this policy, delivers 12.1× lower latency and 5.4× higher throughput than GPU-only computing for OPT-30B inference when both CPU-GPU and GPU-only computing store the model in CPU memory.

Förderung

  • IBM IIDAI and Google

Zitieren

APA

Kim, H., Ye, G., Wang, N., Yazdanbakhsh, A., & Kim, N. S. (2024). Exploiting Intel Advanced Matrix Extensions (AMX) for Large Language Model Inference. IEEE Computer Architecture Letters, 23(1), 117-120. https://doi.org/10.1109/lca.2024.3397747

Vancouver

Kim H, Ye G, Wang N, Yazdanbakhsh A, Kim NS. Exploiting Intel Advanced Matrix Extensions (AMX) for Large Language Model Inference. IEEE Computer Architecture Letters. 2024;23(1):117-120. doi:10.1109/lca.2024.3397747

BibTeX
@article{Kim2024,
  author = {Kim, Hyungyo and Ye, Gaohan and Wang, Nachuan and Yazdanbakhsh, Amir and Kim, Nam Sung},
  title = {Exploiting Intel Advanced Matrix Extensions (AMX) for Large Language Model Inference},
  journal = {IEEE Computer Architecture Letters},
  year = {2024},
  volume = {23},
  number = {1},
  pages = {117--120},
  doi = {10.1109/lca.2024.3397747}
}
Kostenlos starten →