EUVIMEDEUVIMED entdecken
Crossref

Cross-Layer Head Prediction and KV-Cache Prefetching for High-Performance MoH LLM

Yingnan Zhao, Ke Wang, Ahmed Louri, Avinash Karanth, Razvan Bunescu

IEEE Computer Architecture Letters · 2026-07 · 25 (2) · 440-443

DOI: 10.1109/lca.2026.3745257 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Computer Architecture Letters, 2026-07. Verfasst von Yingnan Zhao und 4 weiteren Autor:innen. Die Arbeit nennt 19 Literaturangaben. Gefördert von: NSF CCF, NSF CNS.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Mixture-of-Heads (MoH) attention improves the computational efficiency of large language models (LLMs) by dynamically routing each input token to a subset of attention heads, thereby reducing the number of activated heads and associated on-chip computations. However, MoH-based LLMs still suffer from a memory-latency bottleneck during autoregressive decoding, as the continuously growing key-value (KV) cache requires significant off-chip memory access, substantially increasing latency and energy consumption. To this end, we first characterize the runtime head-activation behavior of MoH-based LLMs and identify stable cross-layer head co-activation patterns across input tokens and representative tasks. Based on this insight, we propose MoHPred, a cross-layer prediction architecture that predicts future activated heads before the corresponding attention computation is executed. By using the predicted heads to proactively fetch the required KV-cache blocks from off-chip memory, MoHPred hides KV-cache loading latency and reduces decoding stalls while preserving the inference accuracy. Evaluation results on real-world applications show that MoHPred reduces execution time by 40% and improves energy efficiency by 69% compared to conventional PyG-GPU and PyG-TPU, demonstrating the effectiveness of co-activation-guided prefetching for efficient MoH-based LLM inference acceleration.

Förderung

  • NSF CCF
  • NSF CNS

Zitieren

APA

Zhao, Y., Wang, K., Louri, A., Karanth, A., & Bunescu, R. (2026). Cross-Layer Head Prediction and KV-Cache Prefetching for High-Performance MoH LLM. IEEE Computer Architecture Letters, 25(2), 440-443. https://doi.org/10.1109/lca.2026.3745257

Vancouver

Zhao Y, Wang K, Louri A, Karanth A, Bunescu R. Cross-Layer Head Prediction and KV-Cache Prefetching for High-Performance MoH LLM. IEEE Computer Architecture Letters. 2026;25(2):440-443. doi:10.1109/lca.2026.3745257

BibTeX
@article{Zhao2026,
  author = {Zhao, Yingnan and Wang, Ke and Louri, Ahmed and Karanth, Avinash and Bunescu, Razvan},
  title = {Cross-Layer Head Prediction and KV-Cache Prefetching for High-Performance MoH LLM},
  journal = {IEEE Computer Architecture Letters},
  year = {2026},
  volume = {25},
  number = {2},
  pages = {440--443},
  doi = {10.1109/lca.2026.3745257}
}
Kostenlos starten →