EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Human-Rated Quality and Measured Token Use in Organizational RAG: A Field Comparison of Three Configurations

Mihael Škarabot, Matjaž Gams, Robert Leskovar

Applied Sciences · 19.11.2026 · 16 (22) · 11481

DOI: 10.3390/app162211481 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 19.11.2026. Verfasst von Mihael Škarabot und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Retrieval-augmented generation (RAG) is commonly assessed on benchmarks, but its behavior under employees’ workplace questions remains less clear. We compared single-pass, agentic, and hybrid RAG in a Slovenian public-sector assistant to examine user-perceived quality, actual tool use, and token consumption. Configurations were randomized per turn, and their names were not shown to users. We analyzed 701 responses from 131 accounts; 402 answers were evaluated by 73 accounts using five 1–5 rating items and a separate binary usefulness judgment. Resource use was application-recorded input plus output tokens, excluding re-ranking. Hybrid had the highest descriptive mean quality (4.15), followed by Single-pass (3.93) and Agentic (3.79), but neither dimension tests nor adjusted composite contrasts established superiority. Hybrid also had the highest mean token total among rated responses (41,143 versus 32,977 and 28,432). Agentic initiated document retrieval on 62.9% of turns; Hybrid used additional retrieval in 3.5%. Thus, the comparison showed a design–behavior gap without establishing an escalation benefit. The study provides field-evaluation evidence and practical deployment recommendations; incomplete rating coverage, configuration-specific prompts, and mixed conversation histories limit generalization and causal interpretation.

Zitieren

APA

Škarabot, M., Gams, M., & Leskovar, R. (2026). Human-Rated Quality and Measured Token Use in Organizational RAG: A Field Comparison of Three Configurations. Applied Sciences, 16(22), 11481. https://doi.org/10.3390/app162211481

Vancouver

Škarabot M, Gams M, Leskovar R. Human-Rated Quality and Measured Token Use in Organizational RAG: A Field Comparison of Three Configurations. Applied Sciences. 2026;16(22):11481. doi:10.3390/app162211481

BibTeX
@article{karabot2026,
  author = {Škarabot, Mihael and Gams, Matjaž and Leskovar, Robert},
  title = {Human-Rated Quality and Measured Token Use in Organizational RAG: A Field Comparison of Three Configurations},
  journal = {Applied Sciences},
  year = {2026},
  volume = {16},
  number = {22},
  pages = {11481},
  doi = {10.3390/app162211481}
}
Kostenlos starten →