Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things
Electronics · 2024-05-27 · 13 (11) · 2077
DOI: 10.3390/electronics13112077 ↗At a glance
Journal article in Electronics, published online on 2024-05-27. Written by Xiaoming Yuan and 3 co-authors. Cited 19 times so far (Crossref); 45 references. The full text is freely available (open access, CC BY 4.0). Funded by: National Natural Science Foundation of China, Science and Technology Project of Hebei Province Education Department, Project of Hebei Key Laboratory of Software Engineering.
Abstract (original language)
Despite recent significant advancements in large language models (LLMs) for medical services, the deployment difficulties of LLMs in e-healthcare hinder complex medical applications in the Internet of Medical Things (IoMT). People are increasingly concerned about e-healthcare risks and privacy protection. Existing LLMs face difficulties in providing accurate medical questions and answers (Q&As) and meeting the deployment resource demands in the IoMT. To address these challenges, we propose MedMixtral 8x7B, a new medical LLM based on the mixture-of-experts (MoE) architecture with an offloading strategy, enabling deployment on the IoMT, improving the privacy protection for users. Additionally, we find that the significant factors affecting latency include the method of device interconnection, the location of offloading servers, and the speed of the disk.
Funding
- National Natural Science Foundation of China
- Science and Technology Project of Hebei Province Education Department
- Project of Hebei Key Laboratory of Software Engineering
Cite
APA
Yuan, X., Kong, W., Luo, Z., & Xu, M. (2024). Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things. Electronics, 13(11), 2077. https://doi.org/10.3390/electronics13112077
Vancouver
Yuan X, Kong W, Luo Z, Xu M. Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things. Electronics. 2024;13(11):2077. doi:10.3390/electronics13112077
BibTeX
@article{Yuan2024,
author = {Yuan, Xiaoming and Kong, Weixuan and Luo, Zhenyu and Xu, Minrui},
title = {Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things},
journal = {Electronics},
year = {2024},
volume = {13},
number = {11},
pages = {2077},
doi = {10.3390/electronics13112077}
}
EUVIMED