Two-Stage Expert Offloading for Domain-Aware MoE Inference
IEEE Access · 2026 · 14 · 33610-33624
DOI: 10.1109/access.2026.3665697 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Access, 2026. Verfasst von Hangyeol Kim und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by Korean Government (MSIT), Development of an On-Device Int…, National Research Foundation of Korea (NRF) grant funded by Korean Government.
Abstract (Originalsprache)
Mixture-of-Experts (MoE) architectures, while computationally efficient, suffer from a critical memory–latency bottleneck. The need to store all experts in GPU memory or to incur high I/O latency when offloading them to CPU severely hinders their practical deployment. To address this, we propose ADEPT (Adaptive Domain-aware Expert Prefetching Technique), a framework designed to optimize expert management during LLM inference under memory-constrained GPU environments. ADEPT employs a two-stage strategy. First, during the prefill phase, it analyzes the input’s semantic domain to selectively preload only relevant experts, minimizing peak memory usage. Second, during token-by-token decoding, a locality-aware mechanism predicts and preemptively loads upcoming experts into the GPU cache, effectively hiding I/O latency. Evaluated across diverse domains, ADEPT demonstrates significant efficiency gains in two key aspects: 1) in terms of latency, simulation results show a 50% reduction in I/O wait time, while real-world validation confirms a $3.45\times $ speedup over standard offloading baselines by mitigating system overheads and 2) in terms of memory, it yields a 33% decrease in peak usage compared to a full-loading approach. Our findings demonstrate that this principled approach significantly improves the practical deployability of MoE models on cost-effective hardware, making them a more viable option for latency-sensitive applications.
Förderung
- Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by Korean Government (MSIT), Development of an On-Device Int…
- National Research Foundation of Korea (NRF) grant funded by Korean Government
Zitieren
APA
Kim, H., Woo, H., & Kim, Y. (2026). Two-Stage Expert Offloading for Domain-Aware MoE Inference. IEEE Access, 14, 33610-33624. https://doi.org/10.1109/access.2026.3665697
Vancouver
Kim H, Woo H, Kim Y. Two-Stage Expert Offloading for Domain-Aware MoE Inference. IEEE Access. 2026;14:33610-33624. doi:10.1109/access.2026.3665697
BibTeX
@article{Kim2026,
author = {Kim, Hangyeol and Woo, Honguk and Kim, Younghwan},
title = {Two-Stage Expert Offloading for Domain-Aware MoE Inference},
journal = {IEEE Access},
year = {2026},
volume = {14},
pages = {33610--33624},
doi = {10.1109/access.2026.3665697}
}
EUVIMED