EUVIMEDOpdag EUVIMED
Open access · CC BY 4.0

Benchmarking Large Language Model Inference on Limited-Resource Edge Systems

Henrikas Giedra, Dalius Matuzevičius, Tomyslav Sledevič, Giga Shubitidze, Artūras Serackis

Electronics · 2026-06-03 · 15 (11) · 2451

DOI: 10.3390/electronics15112451 ↗

At a glance

Journal article in Electronics, published online on 2026-06-03. Written by Henrikas Giedra and 4 co-authors. Cited 4 times so far (Crossref); 77 references. The full text is freely available (open access, CC BY 4.0). Funded by: European Union.

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Large language models (LLMs) are increasingly considered for deployment on edge and limited-resource systems, where local inference can reduce latency, improve privacy, and decrease dependence on cloud infrastructure. While prior studies have evaluated either task accuracy or hardware efficiency in isolation, few benchmarks combine generation-based response-quality evaluation with real-device power measurements on a representative limited-resource platform. This study addresses that gap by benchmarking twelve compact and mid-scale open-weight LLMs (sub-1B to 8B parameters), evaluating generation-based accuracy on a desktop platform and measuring deployment efficiency—throughput, power consumption, and energy use—on an NVIDIA Jetson Orin Nano Super; the accuracy–efficiency trade-off is thus established at the model-configuration level. Unlike prior Jetson-based evaluations relying solely on internal telemetry, this work pairs generation-compatible lm-eval accuracy tasks with a dual power-measurement setup that combines internal tegrastats rail readings with external board-level input power measured using a digital multimeter and explicitly compares GPU-accelerated and CPU-only inference modes. GPU-accelerated inference provided a clear advantage, increasing median throughput from 7.12 to 18.13 tok/s and improving external-meter energy efficiency from 0.453 to 0.823 tok/J, despite higher mean input power. Sub-1B models offered the best throughput and energy efficiency, whereas 7–8B models achieved stronger accuracy at a substantially higher energy cost per generated token. These results demonstrate that edge LLM deployment requires multi-objective evaluation balancing accuracy, throughput, power consumption, and energy efficiency.

Funding

  • European Union

Cite

APA

Giedra, H., Matuzevičius, D., Sledevič, T., Shubitidze, G., & Serackis, A. (2026). Benchmarking Large Language Model Inference on Limited-Resource Edge Systems. Electronics, 15(11), 2451. https://doi.org/10.3390/electronics15112451

Vancouver

Giedra H, Matuzevičius D, Sledevič T, Shubitidze G, Serackis A. Benchmarking Large Language Model Inference on Limited-Resource Edge Systems. Electronics. 2026;15(11):2451. doi:10.3390/electronics15112451

BibTeX
@article{Giedra2026,
  author = {Giedra, Henrikas and Matuzevičius, Dalius and Sledevič, Tomyslav and Shubitidze, Giga and Serackis, Artūras},
  title = {Benchmarking Large Language Model Inference on Limited-Resource Edge Systems},
  journal = {Electronics},
  year = {2026},
  volume = {15},
  number = {11},
  pages = {2451},
  doi = {10.3390/electronics15112451}
}
Start for free →