EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Systematic Evaluation of Large Language Models and Retrieval-Augmented Generation for the Task of Kazakh Question Answering

Aigerim Mansurova, Arailym Tleubayeva, Aliya Nugumanova, Adai Shomanov, Sadi Evren Seker

Information · 30.10.2025 · 16 (11) · 943

DOI: 10.3390/info16110943 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Information, online erschienen am 30.10.2025. Verfasst von Aigerim Mansurova und 4 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 70 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Science Committee of the Ministry of Science and Higher Education of the Republic of Kazakhstan.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper presents a systematic evaluation of large language models (LLMs) and retrieval-augmented generation (RAG) approaches for question answering (QA) in the low-resource Kazakh language. We assess the performance of existing proprietary (GPT-4o, Gemini 2.5-flash) and open-source Kazakh-oriented models (KazLLM-8B, Sherkala-8B, Irbis-7B) across closed-book and RAG settings. Within a three-stage evaluation framework we benchmark retriever quality, examine LLM abilities such as knowledge-gap detection, external truth integration and context grounding, and measures gains from realistic end-to-end RAG pipelines. Our results show a clear pattern: proprietary models lead in closed-book QA, but RAG narrows the gap substantially. Under the Ideal RAG setting, KazLLM-8B improves from its closed-book baseline of 0.427 to reach answer correctness of 0.867, closely matching GPT-4o’s score of 0.869. In the end-to-end RAG setup, KazLLM-8B paired with Snowflake retriever achieved answer correctness up to 0.754, surpassing GPT-4o’s best score of 0.632. Despite improvements, RAG outcomes show an inconsistency: high retrieval metrics do not guarantee high QA system accuracy. The findings highlight the importance of retrievers and context grounding strategies in enabling open-source Kazakh models to deliver competitive QA performance in a low-resource setting.

Förderung

  • Science Committee of the Ministry of Science and Higher Education of the Republic of Kazakhstan

Zitieren

APA

Mansurova, A., Tleubayeva, A., Nugumanova, A., Shomanov, A., & Seker, S. E. (2025). A Systematic Evaluation of Large Language Models and Retrieval-Augmented Generation for the Task of Kazakh Question Answering. Information, 16(11), 943. https://doi.org/10.3390/info16110943

Vancouver

Mansurova A, Tleubayeva A, Nugumanova A, Shomanov A, Seker SE. A Systematic Evaluation of Large Language Models and Retrieval-Augmented Generation for the Task of Kazakh Question Answering. Information. 2025;16(11):943. doi:10.3390/info16110943

BibTeX
@article{Mansurova2025,
  author = {Mansurova, Aigerim and Tleubayeva, Arailym and Nugumanova, Aliya and Shomanov, Adai and Seker, Sadi Evren},
  title = {A Systematic Evaluation of Large Language Models and Retrieval-Augmented Generation for the Task of Kazakh Question Answering},
  journal = {Information},
  year = {2025},
  volume = {16},
  number = {11},
  pages = {943},
  doi = {10.3390/info16110943}
}
Kostenlos starten →