EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Two-Stage Expert Offloading for Domain-Aware MoE Inference

Hangyeol Kim, Honguk Woo, Younghwan Kim

IEEE Access · 2026 · 14 · 33610-33624

DOI: 10.1109/access.2026.3665697 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Access, 2026. Verfasst von Hangyeol Kim und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by Korean Government (MSIT), Development of an On-Device Int…, National Research Foundation of Korea (NRF) grant funded by Korean Government.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Mixture-of-Experts (MoE) architectures, while computationally efficient, suffer from a critical memory–latency bottleneck. The need to store all experts in GPU memory or to incur high I/O latency when offloading them to CPU severely hinders their practical deployment. To address this, we propose ADEPT (Adaptive Domain-aware Expert Prefetching Technique), a framework designed to optimize expert management during LLM inference under memory-constrained GPU environments. ADEPT employs a two-stage strategy. First, during the prefill phase, it analyzes the input’s semantic domain to selectively preload only relevant experts, minimizing peak memory usage. Second, during token-by-token decoding, a locality-aware mechanism predicts and preemptively loads upcoming experts into the GPU cache, effectively hiding I/O latency. Evaluated across diverse domains, ADEPT demonstrates significant efficiency gains in two key aspects: 1) in terms of latency, simulation results show a 50% reduction in I/O wait time, while real-world validation confirms a $3.45\times $ speedup over standard offloading baselines by mitigating system overheads and 2) in terms of memory, it yields a 33% decrease in peak usage compared to a full-loading approach. Our findings demonstrate that this principled approach significantly improves the practical deployability of MoE models on cost-effective hardware, making them a more viable option for latency-sensitive applications.

Förderung

  • Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by Korean Government (MSIT), Development of an On-Device Int…
  • National Research Foundation of Korea (NRF) grant funded by Korean Government

Zitieren

APA

Kim, H., Woo, H., & Kim, Y. (2026). Two-Stage Expert Offloading for Domain-Aware MoE Inference. IEEE Access, 14, 33610-33624. https://doi.org/10.1109/access.2026.3665697

Vancouver

Kim H, Woo H, Kim Y. Two-Stage Expert Offloading for Domain-Aware MoE Inference. IEEE Access. 2026;14:33610-33624. doi:10.1109/access.2026.3665697

BibTeX
@article{Kim2026,
  author = {Kim, Hangyeol and Woo, Honguk and Kim, Younghwan},
  title = {Two-Stage Expert Offloading for Domain-Aware MoE Inference},
  journal = {IEEE Access},
  year = {2026},
  volume = {14},
  pages = {33610--33624},
  doi = {10.1109/access.2026.3665697}
}
Kostenlos starten →