EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Benchmarking Large Language Model Inference on Limited-Resource Edge Systems

Henrikas Giedra, Dalius Matuzevičius, Tomyslav Sledevič, Giga Shubitidze, Artūras Serackis

Electronics · 03.06.2026 · 15 (11) · 2451

DOI: 10.3390/electronics15112451 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 03.06.2026. Verfasst von Henrikas Giedra und 4 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 77 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: European Union.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) are increasingly considered for deployment on edge and limited-resource systems, where local inference can reduce latency, improve privacy, and decrease dependence on cloud infrastructure. While prior studies have evaluated either task accuracy or hardware efficiency in isolation, few benchmarks combine generation-based response-quality evaluation with real-device power measurements on a representative limited-resource platform. This study addresses that gap by benchmarking twelve compact and mid-scale open-weight LLMs (sub-1B to 8B parameters), evaluating generation-based accuracy on a desktop platform and measuring deployment efficiency—throughput, power consumption, and energy use—on an NVIDIA Jetson Orin Nano Super; the accuracy–efficiency trade-off is thus established at the model-configuration level. Unlike prior Jetson-based evaluations relying solely on internal telemetry, this work pairs generation-compatible lm-eval accuracy tasks with a dual power-measurement setup that combines internal tegrastats rail readings with external board-level input power measured using a digital multimeter and explicitly compares GPU-accelerated and CPU-only inference modes. GPU-accelerated inference provided a clear advantage, increasing median throughput from 7.12 to 18.13 tok/s and improving external-meter energy efficiency from 0.453 to 0.823 tok/J, despite higher mean input power. Sub-1B models offered the best throughput and energy efficiency, whereas 7–8B models achieved stronger accuracy at a substantially higher energy cost per generated token. These results demonstrate that edge LLM deployment requires multi-objective evaluation balancing accuracy, throughput, power consumption, and energy efficiency.

Förderung

  • European Union

Zitieren

APA

Giedra, H., Matuzevičius, D., Sledevič, T., Shubitidze, G., & Serackis, A. (2026). Benchmarking Large Language Model Inference on Limited-Resource Edge Systems. Electronics, 15(11), 2451. https://doi.org/10.3390/electronics15112451

Vancouver

Giedra H, Matuzevičius D, Sledevič T, Shubitidze G, Serackis A. Benchmarking Large Language Model Inference on Limited-Resource Edge Systems. Electronics. 2026;15(11):2451. doi:10.3390/electronics15112451

BibTeX
@article{Giedra2026,
  author = {Giedra, Henrikas and Matuzevičius, Dalius and Sledevič, Tomyslav and Shubitidze, Giga and Serackis, Artūras},
  title = {Benchmarking Large Language Model Inference on Limited-Resource Edge Systems},
  journal = {Electronics},
  year = {2026},
  volume = {15},
  number = {11},
  pages = {2451},
  doi = {10.3390/electronics15112451}
}
Infokarte | EUVIMED

Artikeldaten derzeit nicht verfügbar.

Recherche