EUVIMEDEUVIMED entdecken
Open Access · CC BY-ND 4.0

Evaluating RAG Performance on Small Language Models for Low-Resource Devices through Chunking and Retrieval Methods

Amelia Dewi Agustiani, Salsabila Maharani Putri, Jonner Hutahaean, Muhammad Rizqi Sholahuddin, Muhammad Riza Alifi, Ade Hodijah

Jurnal Online Informatika · 14.05.2026 · 11 (1) · 243-255

DOI: 10.15575/join.v11i1.1733 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Jurnal Online Informatika, online erschienen am 14.05.2026. Verfasst von Amelia Dewi Agustiani und 5 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Retrieval-Augmented Generation (RAG) combines generative capabilities of language models with external document retrieval to answer questions grounded in reference texts. However, deploying RAG on low-resource devices like Android smartphones is challenging because SLMs have limited computational capacity and depend heavily on efficient chunking and retrieval. Although interest in on-device processing is growing, research on RAG configurations for SLMs under strict resource constraints especially for domain-specific tasks remains limited. This study therefore investigates which combinations of chunking technique, chunk size, overlap, and retrieval strategy best balance accuracy and speed on low-resource devices. The evaluation uses 148 Indonesian questions sourced from an official Hajj guidebook. The study consists of two phases retrieval and generation. Retrieval is evaluated using BLEU, ROUGE-L, MRR, MAP, and Hit@k, while answer quality is measured with BERTScore. The experiments compare different chunking methods (fixed-size or semantic), chunk sizes (128 or 256 tokens), overlaps (25, 50 and 100 tokens), and retrieval methods (dense, sparse, or hybrid). Results show that sparse retrieval with 256-token chunks and 100-token overlap yields the best answer quality (F1 = 0.726). However, 128-token chunks with the same overlap provide the fastest generation time (69.737 seconds). The main contribution of this study is a systematic evaluation of RAG configurations for fully on-device SLMs using a domain-specific Hajj and Umrah dataset not explored in prior research. The findings provide practical guidance for designing efficient and accurate RAG-based question-answering systems on low-resource devices.

Zitieren

APA

Agustiani, A. D., Putri, S. M., Hutahaean, J., Sholahuddin, M. R., Alifi, M. R., & Hodijah, A. (2026). Evaluating RAG Performance on Small Language Models for Low-Resource Devices through Chunking and Retrieval Methods. Jurnal Online Informatika, 11(1), 243-255. https://doi.org/10.15575/join.v11i1.1733

Vancouver

Agustiani AD, Putri SM, Hutahaean J, Sholahuddin MR, Alifi MR, Hodijah A. Evaluating RAG Performance on Small Language Models for Low-Resource Devices through Chunking and Retrieval Methods. Jurnal Online Informatika. 2026;11(1):243-255. doi:10.15575/join.v11i1.1733

BibTeX
@article{Agustiani2026,
  author = {Agustiani, Amelia Dewi and Putri, Salsabila Maharani and Hutahaean, Jonner and Sholahuddin, Muhammad Rizqi and Alifi, Muhammad Riza and Hodijah, Ade},
  title = {Evaluating RAG Performance on Small Language Models for Low-Resource Devices through Chunking and Retrieval Methods},
  journal = {Jurnal Online Informatika},
  year = {2026},
  volume = {11},
  number = {1},
  pages = {243--255},
  doi = {10.15575/join.v11i1.1733}
}
Kostenlos starten →