Sometimes Painful but Promising: Feasibility and Trade-Offs of On-Device Language Model Inference
ACM Transactions on Embedded Computing Systems · 31.03.2026 · 25 (2) · 1-30
DOI: 10.1145/3788870 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Embedded Computing Systems, online erschienen am 03.03.2026. Verfasst von Maximilian Abstreiter und 2 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 61 Literaturangaben. Gefördert von: Business Finland 6G Bridge Program, NordForsk.
Abstract (Originalsprache)
The rapid rise of Language Models (LMs) has expanded the capabilities of natural language processing, powering applications from text generation to complex decision-making. While state-of-the-art LMs often boast hundreds of billions of parameters and are primarily deployed in data centers, recent trends show a growing focus on compact models—typically under 10 billion parameters–enabled by techniques such as quantization and other model compression techniques. This shift paves the way for LMs on edge devices, offering potential benefits such as enhanced privacy, reduced latency, and improved data sovereignty. However, the inherent complexity of even these smaller models, combined with the limited computing resources of edge hardware, raises critical questions about the practical trade-offs in executing LM inference outside the cloud. To address these challenges, we present a comprehensive evaluation of generative LM inference on representative CPU-based and GPU-accelerated edge devices. Our study measures key performance indicators—including memory usage, inference speed, and energy consumption—across various device configurations. Additionally, we examine throughput-energy trade-offs, cost considerations, and usability, alongside an assessment of qualitative model performance. While quantization helps mitigate memory overhead, it does not fully eliminate resource bottlenecks, especially for larger models. Our findings quantify the memory and energy constraints that must be considered for practical real-world deployments, offering concrete insights into the trade-offs between model size, inference performance, and efficiency. The exploration of LMs at the edge is still in its early stages. We hope this study provides a foundation for future research, guiding the refinement of models, the enhancement of inference efficiency, and the advancement of edge-centric AI systems.
Förderung
- Business Finland 6G Bridge Program
- NordForsk
Zitieren
APA
Abstreiter, M., Tarkoma, S., & Morabito, R. (2026). Sometimes Painful but Promising: Feasibility and Trade-Offs of On-Device Language Model Inference. ACM Transactions on Embedded Computing Systems, 25(2), 1-30. https://doi.org/10.1145/3788870
Vancouver
Abstreiter M, Tarkoma S, Morabito R. Sometimes Painful but Promising: Feasibility and Trade-Offs of On-Device Language Model Inference. ACM Transactions on Embedded Computing Systems. 2026;25(2):1-30. doi:10.1145/3788870
BibTeX
@article{Abstreiter2026,
author = {Abstreiter, Maximilian and Tarkoma, Sasu and Morabito, Roberto},
title = {Sometimes Painful but Promising: Feasibility and Trade-Offs of On-Device Language Model Inference},
journal = {ACM Transactions on Embedded Computing Systems},
year = {2026},
volume = {25},
number = {2},
pages = {1--30},
doi = {10.1145/3788870}
}
EUVIMED