Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things
Electronics · 27.05.2024 · 13 (11) · 2077
DOI: 10.3390/electronics13112077 ↗Kurz gefasst
Fachartikel in der Zeitschrift Electronics, online erschienen am 27.05.2024. Verfasst von Xiaoming Yuan und 3 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Science and Technology Project of Hebei Province Education Department, Project of Hebei Key Laboratory of Software Engineering.
Abstract (Originalsprache)
Despite recent significant advancements in large language models (LLMs) for medical services, the deployment difficulties of LLMs in e-healthcare hinder complex medical applications in the Internet of Medical Things (IoMT). People are increasingly concerned about e-healthcare risks and privacy protection. Existing LLMs face difficulties in providing accurate medical questions and answers (Q&As) and meeting the deployment resource demands in the IoMT. To address these challenges, we propose MedMixtral 8x7B, a new medical LLM based on the mixture-of-experts (MoE) architecture with an offloading strategy, enabling deployment on the IoMT, improving the privacy protection for users. Additionally, we find that the significant factors affecting latency include the method of device interconnection, the location of offloading servers, and the speed of the disk.
Förderung
- National Natural Science Foundation of China
- Science and Technology Project of Hebei Province Education Department
- Project of Hebei Key Laboratory of Software Engineering
Zitieren
APA
Yuan, X., Kong, W., Luo, Z., & Xu, M. (2024). Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things. Electronics, 13(11), 2077. https://doi.org/10.3390/electronics13112077
Vancouver
Yuan X, Kong W, Luo Z, Xu M. Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things. Electronics. 2024;13(11):2077. doi:10.3390/electronics13112077
BibTeX
@article{Yuan2024,
author = {Yuan, Xiaoming and Kong, Weixuan and Luo, Zhenyu and Xu, Minrui},
title = {Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things},
journal = {Electronics},
year = {2024},
volume = {13},
number = {11},
pages = {2077},
doi = {10.3390/electronics13112077}
}
EUVIMED