EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

RAG Effectiveness Across Model Size and Architecture: A Cross-Family Evaluation on a Domain-Specific Croatian Accounting Benchmark

Kristian Dokic, Marko Martinovic, Mario Zupan, Mirjana Radman-Funaric

Applied Sciences · 03.11.2026 · 16 (21) · 10901

DOI: 10.3390/app162110901 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 03.11.2026. Verfasst von Kristian Dokic und 3 weiteren Autor:innen. Die Arbeit nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Recovery and Resilience Plan (NPOO) of the Republic of Croatia.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper presents a systematic empirical evaluation of Retrieval-Augmented Generation (RAG) across 22 open-weight language models spanning five families—Qwen3.5, Llama 3.x, Gemma 3, Phi-4, and DeepSeek R1 distilled—on a domain-specific Croatian accounting benchmark derived from the publicly available mariozupan/rrif dataset. All models were deployed locally via Ollama on a single GPU server, reflecting privacy-preserving enterprise conditions. Two retrieval corpora were evaluated: a structured English Q&A-derived corpus (Corpus A, 522 accounts) and a Croatian corpus parsed from the RRiF chart of accounts (Corpus B, 2462 entries), the latter indexed under three embedding models, yielding five conditions and 116,600 queries in total. Four findings are reported. First, on Corpus A, RAG raises lookup Exact Match from near-zero to 94.9–96.2% with 18 of the 22 models converging into that band, with equivalence testing confirming practical equivalence for 87% of within-band pairs at a two-point margin; the four exceptions lie at 1B–3B parameters, but the boundary tracks instruction-following quality rather than scale. Because Corpus A shares its source with the evaluation questions, these results are an upper bound rather than an estimate of field performance. Second, embedding model choice dominates language model choice on the Croatian corpus, where a multilingual encoder raises average Exact Match by approximately 43 percentage points. Third, reasoning-distilled models show no consistent advantage on this task. Fourth, decomposing responses into mutually exclusive outcome categories shows that retrieval reduces the invention of account codes from 33.6% to below 0.3% under every condition tested, while residual error consists almost entirely of wrong-but-existing accounts whose source shifts from retrieval to selection as retrieval improves—a distinction that a binary hallucination measure conflates. Non-generative baselines outperform every model on Corpus A and ten of the 22 on Corpus B, indicating that a language model is warranted only conditionally for this task.

Förderung

  • National Recovery and Resilience Plan (NPOO) of the Republic of Croatia

Zitieren

APA

Dokic, K., Martinovic, M., Zupan, M., & Radman-Funaric, M. (2026). RAG Effectiveness Across Model Size and Architecture: A Cross-Family Evaluation on a Domain-Specific Croatian Accounting Benchmark. Applied Sciences, 16(21), 10901. https://doi.org/10.3390/app162110901

Vancouver

Dokic K, Martinovic M, Zupan M, Radman-Funaric M. RAG Effectiveness Across Model Size and Architecture: A Cross-Family Evaluation on a Domain-Specific Croatian Accounting Benchmark. Applied Sciences. 2026;16(21):10901. doi:10.3390/app162110901

BibTeX
@article{Dokic2026,
  author = {Dokic, Kristian and Martinovic, Marko and Zupan, Mario and Radman-Funaric, Mirjana},
  title = {RAG Effectiveness Across Model Size and Architecture: A Cross-Family Evaluation on a Domain-Specific Croatian Accounting Benchmark},
  journal = {Applied Sciences},
  year = {2026},
  volume = {16},
  number = {21},
  pages = {10901},
  doi = {10.3390/app162110901}
}
Kostenlos starten →