EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things

Xiaoming Yuan, Weixuan Kong, Zhenyu Luo, Minrui Xu

Electronics · 27.05.2024 · 13 (11) · 2077

DOI: 10.3390/electronics13112077 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 27.05.2024. Verfasst von Xiaoming Yuan und 3 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Science and Technology Project of Hebei Province Education Department, Project of Hebei Key Laboratory of Software Engineering.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Despite recent significant advancements in large language models (LLMs) for medical services, the deployment difficulties of LLMs in e-healthcare hinder complex medical applications in the Internet of Medical Things (IoMT). People are increasingly concerned about e-healthcare risks and privacy protection. Existing LLMs face difficulties in providing accurate medical questions and answers (Q&As) and meeting the deployment resource demands in the IoMT. To address these challenges, we propose MedMixtral 8x7B, a new medical LLM based on the mixture-of-experts (MoE) architecture with an offloading strategy, enabling deployment on the IoMT, improving the privacy protection for users. Additionally, we find that the significant factors affecting latency include the method of device interconnection, the location of offloading servers, and the speed of the disk.

Förderung

  • National Natural Science Foundation of China
  • Science and Technology Project of Hebei Province Education Department
  • Project of Hebei Key Laboratory of Software Engineering

Zitieren

APA

Yuan, X., Kong, W., Luo, Z., & Xu, M. (2024). Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things. Electronics, 13(11), 2077. https://doi.org/10.3390/electronics13112077

Vancouver

Yuan X, Kong W, Luo Z, Xu M. Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things. Electronics. 2024;13(11):2077. doi:10.3390/electronics13112077

BibTeX
@article{Yuan2024,
  author = {Yuan, Xiaoming and Kong, Weixuan and Luo, Zhenyu and Xu, Minrui},
  title = {Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things},
  journal = {Electronics},
  year = {2024},
  volume = {13},
  number = {11},
  pages = {2077},
  doi = {10.3390/electronics13112077}
}
Kostenlos starten →