Human-Rated Quality and Measured Token Use in Organizational RAG: A Field Comparison of Three Configurations
Applied Sciences · 19.11.2026 · 16 (22) · 11481
DOI: 10.3390/app162211481 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 19.11.2026. Verfasst von Mihael Škarabot und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Retrieval-augmented generation (RAG) is commonly assessed on benchmarks, but its behavior under employees’ workplace questions remains less clear. We compared single-pass, agentic, and hybrid RAG in a Slovenian public-sector assistant to examine user-perceived quality, actual tool use, and token consumption. Configurations were randomized per turn, and their names were not shown to users. We analyzed 701 responses from 131 accounts; 402 answers were evaluated by 73 accounts using five 1–5 rating items and a separate binary usefulness judgment. Resource use was application-recorded input plus output tokens, excluding re-ranking. Hybrid had the highest descriptive mean quality (4.15), followed by Single-pass (3.93) and Agentic (3.79), but neither dimension tests nor adjusted composite contrasts established superiority. Hybrid also had the highest mean token total among rated responses (41,143 versus 32,977 and 28,432). Agentic initiated document retrieval on 62.9% of turns; Hybrid used additional retrieval in 3.5%. Thus, the comparison showed a design–behavior gap without establishing an escalation benefit. The study provides field-evaluation evidence and practical deployment recommendations; incomplete rating coverage, configuration-specific prompts, and mixed conversation histories limit generalization and causal interpretation.
Zitieren
APA
Škarabot, M., Gams, M., & Leskovar, R. (2026). Human-Rated Quality and Measured Token Use in Organizational RAG: A Field Comparison of Three Configurations. Applied Sciences, 16(22), 11481. https://doi.org/10.3390/app162211481
Vancouver
Škarabot M, Gams M, Leskovar R. Human-Rated Quality and Measured Token Use in Organizational RAG: A Field Comparison of Three Configurations. Applied Sciences. 2026;16(22):11481. doi:10.3390/app162211481
BibTeX
@article{karabot2026,
author = {Škarabot, Mihael and Gams, Matjaž and Leskovar, Robert},
title = {Human-Rated Quality and Measured Token Use in Organizational RAG: A Field Comparison of Three Configurations},
journal = {Applied Sciences},
year = {2026},
volume = {16},
number = {22},
pages = {11481},
doi = {10.3390/app162211481}
}
EUVIMED