EUVIMEDOpdag EUVIMED
Open access · CC BY 4.0

Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things

Xiaoming Yuan, Weixuan Kong, Zhenyu Luo, Minrui Xu

Electronics · 2024-05-27 · 13 (11) · 2077

DOI: 10.3390/electronics13112077 ↗

At a glance

Journal article in Electronics, published online on 2024-05-27. Written by Xiaoming Yuan and 3 co-authors. Cited 19 times so far (Crossref); 45 references. The full text is freely available (open access, CC BY 4.0). Funded by: National Natural Science Foundation of China, Science and Technology Project of Hebei Province Education Department, Project of Hebei Key Laboratory of Software Engineering.

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Despite recent significant advancements in large language models (LLMs) for medical services, the deployment difficulties of LLMs in e-healthcare hinder complex medical applications in the Internet of Medical Things (IoMT). People are increasingly concerned about e-healthcare risks and privacy protection. Existing LLMs face difficulties in providing accurate medical questions and answers (Q&As) and meeting the deployment resource demands in the IoMT. To address these challenges, we propose MedMixtral 8x7B, a new medical LLM based on the mixture-of-experts (MoE) architecture with an offloading strategy, enabling deployment on the IoMT, improving the privacy protection for users. Additionally, we find that the significant factors affecting latency include the method of device interconnection, the location of offloading servers, and the speed of the disk.

Funding

  • National Natural Science Foundation of China
  • Science and Technology Project of Hebei Province Education Department
  • Project of Hebei Key Laboratory of Software Engineering

Cite

APA

Yuan, X., Kong, W., Luo, Z., & Xu, M. (2024). Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things. Electronics, 13(11), 2077. https://doi.org/10.3390/electronics13112077

Vancouver

Yuan X, Kong W, Luo Z, Xu M. Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things. Electronics. 2024;13(11):2077. doi:10.3390/electronics13112077

BibTeX
@article{Yuan2024,
  author = {Yuan, Xiaoming and Kong, Weixuan and Luo, Zhenyu and Xu, Minrui},
  title = {Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things},
  journal = {Electronics},
  year = {2024},
  volume = {13},
  number = {11},
  pages = {2077},
  doi = {10.3390/electronics13112077}
}
Start for free →