Obširna evalvacija komercialnih velikih jezikovnih modelov na področju sklepanja v slovenskem jeziku in slovnice
Uporabna informatika · 15.05.2026 · 34 (1)
DOI: 10.31449/upinf.270 ↗Kurz gefasst
Fachartikel in der Zeitschrift Uporabna informatika, online erschienen am 15.05.2026. Verfasst von Miha Malenšek und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
The use of large language models (LLMs) is rapidly expanding in the Slovenian context; however, their actual performance on the Slovenian language remains insufficiently and unsystematically evaluated. In this paper, we present an extensive comparative evaluation of the most widely used commercial and open-source LLMs with respect to Slovenian. The evaluation includes models from four major providers (OpenAI, Google, Anthropic, and Mistral), as well as the domestic models GaMS-27B-Instruct and GaMS3-12B-Instruct, and assesses them using a diverse set of benchmarks targeting instruction following, reasoning abilities, answer reliability, grammatical competence, and textual coherence. We employ translated standardized benchmark tasks (e.g., ARC, HellaSwag, TruthfulQA, GSM8K), the specialized grammatical error dataset DASSLE 1.0, and a collection of real-world conversations from the Slovenian Conversational Arena. The results show that contemporary commercial models achieve high performance on comprehension and reasoning tasks in Slovenian—most notably GPT-5.1 with a high level of deliberative reasoning and Gemini-2.5-Pro—while open models such as Mistral Large 3 attain competitive results despite more limited resources. In contrast, the evaluation of grammatical competence reveals that the morphological and syntactic complexity of Slovenian remains a significant challenge for all evaluated models. Overall, the paper provides a comprehensive overview of the current state of LLM performance for the Slovenian language.
Zitieren
APA
Malenšek, M., Vreš, D., & Bajec, M. (2026). Obširna evalvacija komercialnih velikih jezikovnih modelov na področju sklepanja v slovenskem jeziku in slovnice. Uporabna informatika, 34(1). https://doi.org/10.31449/upinf.270
Vancouver
Malenšek M, Vreš D, Bajec M. Obširna evalvacija komercialnih velikih jezikovnih modelov na področju sklepanja v slovenskem jeziku in slovnice. Uporabna informatika. 2026;34(1). doi:10.31449/upinf.270
BibTeX
@article{Malenek2026,
author = {Malenšek, Miha and Vreš, Domen and Bajec, Marko},
title = {Obširna evalvacija komercialnih velikih jezikovnih modelov na področju sklepanja v slovenskem jeziku in slovnice},
journal = {Uporabna informatika},
year = {2026},
volume = {34},
number = {1},
doi = {10.31449/upinf.270}
}
EUVIMED