EUVIMEDEUVIMED entdecken
Crossref

VectorWeaver: A Stage-Aware Automated Optimization Framework for LLM Inference on Edge Platform

Pengfei Yang, Hui Zeng, Wenxuan Hou, Mingwei Wang, Weiye Ji, Tianyang Zheng, Hui Li

ACM Transactions on Architecture and Code Optimization · 31.03.2026 · 23 (1) · 1-24

DOI: 10.1145/3799719 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Architecture and Code Optimization, online erschienen am 23.03.2026. Verfasst von Pengfei Yang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 48 Literaturangaben. Gefördert von: Natural Science Funds for Distinguished Young Scholar of Shaanxi, Shaanxi Key Technology R&D Program, Zhongguancun Academy Internal Research.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The convergence of Large Language Models (LLMs) and the open-source RISC-V architecture creates significant opportunities for specialized AI hardware. However, achieving optimal LLM inference performance on RISC-V platforms is challenging due to the complex computational patterns of LLM operators and the architectural nuances of the RISC-V Vector (RVV) extension. Existing optimization strategies often rely on manual, expert-driven tuning, which is labor-intensive and not generalizable across diverse operators and hardware configurations. This article presents VectorWeaver, an automated framework to generate high-performance, stage-aware microkernels for LLM inference on RISC-V. VectorWeaver specifically addresses key performance bottlenecks in quantized models, such as memory discontinuities in vector dot-product operations, by employing a widen vectorization strategy. Guided by a stage-aware performance model, the framework systematically explores a rich optimization space encompassing loop unrolling, software pipelining, and mathematical approximations. The model generates distinct, highly-optimized kernels for the prefill and decode stages, selected at runtime via a dispatch mechanism. We evaluate VectorWeaver on the TH1520 and the Sophon SG2044 using Gemma3 and Qwen3 models with sizes ranging from 270M to 4B, where our generated microkernels achieve up to a 24x speedup over baseline implementations. For end-to-end evaluations, VectorWeaver delivers up to a 5.2x increase in inference throughput, demonstrating its effectiveness in automating the generation of high-performance code and advancing the state of LLM inference on the RISC-V ecosystem.

Förderung

  • Natural Science Funds for Distinguished Young Scholar of Shaanxi
  • Shaanxi Key Technology R&D Program
  • Zhongguancun Academy Internal Research

Zitieren

APA

Yang, P., Zeng, H., Hou, W., Wang, M., Ji, W., Zheng, T., & Li, H. (2026). VectorWeaver: A Stage-Aware Automated Optimization Framework for LLM Inference on Edge Platform. ACM Transactions on Architecture and Code Optimization, 23(1), 1-24. https://doi.org/10.1145/3799719

Vancouver

Yang P, Zeng H, Hou W, Wang M, Ji W, Zheng T, et al. VectorWeaver: A Stage-Aware Automated Optimization Framework for LLM Inference on Edge Platform. ACM Transactions on Architecture and Code Optimization. 2026;23(1):1-24. doi:10.1145/3799719

BibTeX
@article{Yang2026,
  author = {Yang, Pengfei and Zeng, Hui and Hou, Wenxuan and Wang, Mingwei and Ji, Weiye and Zheng, Tianyang and Li, Hui},
  title = {VectorWeaver: A Stage-Aware Automated Optimization Framework for LLM Inference on Edge Platform},
  journal = {ACM Transactions on Architecture and Code Optimization},
  year = {2026},
  volume = {23},
  number = {1},
  pages = {1--24},
  doi = {10.1145/3799719}
}
Kostenlos starten →