A Systematic Evaluation of Large Language Models and Retrieval-Augmented Generation for the Task of Kazakh Question Answering
Information · 30.10.2025 · 16 (11) · 943
DOI: 10.3390/info16110943 ↗Kurz gefasst
Fachartikel in der Zeitschrift Information, online erschienen am 30.10.2025. Verfasst von Aigerim Mansurova und 4 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 70 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Science Committee of the Ministry of Science and Higher Education of the Republic of Kazakhstan.
Abstract (Originalsprache)
This paper presents a systematic evaluation of large language models (LLMs) and retrieval-augmented generation (RAG) approaches for question answering (QA) in the low-resource Kazakh language. We assess the performance of existing proprietary (GPT-4o, Gemini 2.5-flash) and open-source Kazakh-oriented models (KazLLM-8B, Sherkala-8B, Irbis-7B) across closed-book and RAG settings. Within a three-stage evaluation framework we benchmark retriever quality, examine LLM abilities such as knowledge-gap detection, external truth integration and context grounding, and measures gains from realistic end-to-end RAG pipelines. Our results show a clear pattern: proprietary models lead in closed-book QA, but RAG narrows the gap substantially. Under the Ideal RAG setting, KazLLM-8B improves from its closed-book baseline of 0.427 to reach answer correctness of 0.867, closely matching GPT-4o’s score of 0.869. In the end-to-end RAG setup, KazLLM-8B paired with Snowflake retriever achieved answer correctness up to 0.754, surpassing GPT-4o’s best score of 0.632. Despite improvements, RAG outcomes show an inconsistency: high retrieval metrics do not guarantee high QA system accuracy. The findings highlight the importance of retrievers and context grounding strategies in enabling open-source Kazakh models to deliver competitive QA performance in a low-resource setting.
Förderung
- Science Committee of the Ministry of Science and Higher Education of the Republic of Kazakhstan
Zitieren
APA
Mansurova, A., Tleubayeva, A., Nugumanova, A., Shomanov, A., & Seker, S. E. (2025). A Systematic Evaluation of Large Language Models and Retrieval-Augmented Generation for the Task of Kazakh Question Answering. Information, 16(11), 943. https://doi.org/10.3390/info16110943
Vancouver
Mansurova A, Tleubayeva A, Nugumanova A, Shomanov A, Seker SE. A Systematic Evaluation of Large Language Models and Retrieval-Augmented Generation for the Task of Kazakh Question Answering. Information. 2025;16(11):943. doi:10.3390/info16110943
BibTeX
@article{Mansurova2025,
author = {Mansurova, Aigerim and Tleubayeva, Arailym and Nugumanova, Aliya and Shomanov, Adai and Seker, Sadi Evren},
title = {A Systematic Evaluation of Large Language Models and Retrieval-Augmented Generation for the Task of Kazakh Question Answering},
journal = {Information},
year = {2025},
volume = {16},
number = {11},
pages = {943},
doi = {10.3390/info16110943}
}
EUVIMED