EUVIMEDEUVIMED entdecken
Crossref

Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency

Erik Johannes Husom, Arda Goknil, Merve Astekin, Lwin Khin Shar, Andre KÃ¥sen, Sagar Sen, Benedikt Andreas Mithassel, Ahmet Soylu

ACM Transactions on Internet of Things · 30.11.2025 · 6 (4) · 1-35

DOI: 10.1145/3767742 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Internet of Things, online erschienen am 18.11.2025. Verfasst von Erik Johannes Husom und 7 weiteren Autor:innen. Die Arbeit wurde bisher 55-mal zitiert (Crossref) und nennt 88 Literaturangaben. Gefördert von: ENFIELD project, NSFEuropean Commission within the HEU Programme, and the National Research Foundation of Korea.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Deploying Large Language Models (LLMs) on edge devices presents significant challenges due to computational constraints, memory limitations, inference speed, and energy consumption. Model quantization has emerged as a key technique to enable efficient LLM inference by reducing model size and computational overhead. In this study, we conduct a comprehensive analysis of 28 quantized LLMs from the Ollama library, which applies by default Post-Training Quantization (PTQ) and weight-only quantization techniques, deployed on an edge device (Raspberry Pi 4 with 4 GB RAM). We evaluate energy efficiency, inference performance, and output accuracy across multiple quantization levels and task types. Models are benchmarked on five standardized datasets (CommonsenseQA, BIG-Bench Hard, TruthfulQA, GSM8K, and HumanEval), and we employ a high-resolution, hardware-based energy measurement tool to capture real-world power consumption. Our findings reveal the trade-offs between energy efficiency, inference speed, and accuracy in different quantization settings, highlighting configurations that optimize LLM deployment for resource-constrained environments. By integrating hardware-level energy profiling with LLM benchmarking, this study provides actionable insights for sustainable AI, bridging a critical gap in existing research on energy-aware LLM deployment.

Förderung

  • ENFIELD project
  • NSFEuropean Commission within the HEU Programme, and the National Research Foundation of Korea

Zitieren

APA

Husom, E. J., Goknil, A., Astekin, M., Shar, L. K., KÃ¥sen, A., Sen, S., Mithassel, B. A., & Soylu, A. (2025). Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency. ACM Transactions on Internet of Things, 6(4), 1-35. https://doi.org/10.1145/3767742

Vancouver

Husom EJ, Goknil A, Astekin M, Shar LK, KÃ¥sen A, Sen S, et al. Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency. ACM Transactions on Internet of Things. 2025;6(4):1-35. doi:10.1145/3767742

BibTeX
@article{Husom2025,
  author = {Husom, Erik Johannes and Goknil, Arda and Astekin, Merve and Shar, Lwin Khin and KÃ¥sen, Andre and Sen, Sagar and Mithassel, Benedikt Andreas and Soylu, Ahmet},
  title = {Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency},
  journal = {ACM Transactions on Internet of Things},
  year = {2025},
  volume = {6},
  number = {4},
  pages = {1--35},
  doi = {10.1145/3767742}
}
Kostenlos starten →