EUVIMEDEUVIMED entdecken
Crossref

Device-Cloud Collaborative LLM Inference With Multi-Modal, Multi-Task, and Multi-Turn Conversations

Liangqi Yuan, Dong-Jun Han, Shiqiang Wang, Christopher G. Brinton

IEEE Transactions on Networking · 2026 · 34 · 6124-6139

DOI: 10.1109/ton.2026.3712685 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Networking, 2026. Verfasst von Liangqi Yuan und 3 weiteren Autor:innen. Die Arbeit nennt 49 Literaturangaben. Gefördert von: National Science Foundation, Office of Naval Research, Air Force Office of Scientific Research.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Compared to traditional machine learning models, recent large language models (LLMs) can exhibit multi-task-solving capabilities through multi-modal data sources and multi-turn conversations. These unique characteristics of LLMs, together with their large model size, make their deployment more challenging. Specifically, 1) deploying LLMs on devices faces computational, memory, and energy resource issues, while 2) deploying them in the cloud cannot guarantee real-time service and incurs communication/usage costs. In this paper, we design TMO, a device-cloud LLM inference system with Three-M Offloading: Multi-modal, Multi-task, and Multi-turn conversation. TMO incorporates 1) a lightweight on-device LLM that can process simple tasks at high speed and 2) a large-scale cloud LLM that can handle multi-modal data sources. We develop a resource-constrained reinforcement learning (RCRL) strategy for TMO that optimizes the inference location (i.e., device vs. cloud) and multi-modal data sources to use for each task in multi-turn conversations, aiming to maximize the long-term reward (response quality, latency, and usage cost) while adhering to resource constraints. We also contribute M4A1, a new dataset we curated across multiple modalities, tasks, conversation turns, and LLM configurations, enabling evaluation of offloading decisions. We demonstrate the effectiveness of TMO compared to several exploration-decision and LLM-as-Router baselines, showing significant improvements in latency, cost, and response quality. Our code and dataset are available at https://github.com/liangqiyuan/TMO

Förderung

  • National Science Foundation
  • Office of Naval Research
  • Air Force Office of Scientific Research
  • Institute of Information and Communications Technology Planning and Evaluation (IITP) from Korean Government [Ministry of Science and Information and Communica…
  • NVIDIA’s Academic Grant Program

Zitieren

APA

Yuan, L., Han, D., Wang, S., & Brinton, C. G. (2026). Device-Cloud Collaborative LLM Inference With Multi-Modal, Multi-Task, and Multi-Turn Conversations. IEEE Transactions on Networking, 34, 6124-6139. https://doi.org/10.1109/ton.2026.3712685

Vancouver

Yuan L, Han D, Wang S, Brinton CG. Device-Cloud Collaborative LLM Inference With Multi-Modal, Multi-Task, and Multi-Turn Conversations. IEEE Transactions on Networking. 2026;34:6124-6139. doi:10.1109/ton.2026.3712685

BibTeX
@article{Yuan2026,
  author = {Yuan, Liangqi and Han, Dong-Jun and Wang, Shiqiang and Brinton, Christopher G.},
  title = {Device-Cloud Collaborative LLM Inference With Multi-Modal, Multi-Task, and Multi-Turn Conversations},
  journal = {IEEE Transactions on Networking},
  year = {2026},
  volume = {34},
  pages = {6124--6139},
  doi = {10.1109/ton.2026.3712685}
}
Kostenlos starten →