EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Obširna evalvacija komercialnih velikih jezikovnih modelov na področju sklepanja v slovenskem jeziku in slovnice

Miha Malenšek, Domen Vreš, Marko Bajec

Uporabna informatika · 15.05.2026 · 34 (1)

DOI: 10.31449/upinf.270 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Uporabna informatika, online erschienen am 15.05.2026. Verfasst von Miha Malenšek und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The use of large language models (LLMs) is rapidly expanding in the Slovenian context; however, their actual performance on the Slovenian language remains insufficiently and unsystematically evaluated. In this paper, we present an extensive comparative evaluation of the most widely used commercial and open-source LLMs with respect to Slovenian. The evaluation includes models from four major providers (OpenAI, Google, Anthropic, and Mistral), as well as the domestic models GaMS-27B-Instruct and GaMS3-12B-Instruct, and assesses them using a diverse set of benchmarks targeting instruction following, reasoning abilities, answer reliability, grammatical competence, and textual coherence. We employ translated standardized benchmark tasks (e.g., ARC, HellaSwag, TruthfulQA, GSM8K), the specialized grammatical error dataset DASSLE 1.0, and a collection of real-world conversations from the Slovenian Conversational Arena. The results show that contemporary commercial models achieve high performance on comprehension and reasoning tasks in Slovenian—most notably GPT-5.1 with a high level of deliberative reasoning and Gemini-2.5-Pro—while open models such as Mistral Large 3 attain competitive results despite more limited resources. In contrast, the evaluation of grammatical competence reveals that the morphological and syntactic complexity of Slovenian remains a significant challenge for all evaluated models. Overall, the paper provides a comprehensive overview of the current state of LLM performance for the Slovenian language.

Zitieren

APA

Malenšek, M., Vreš, D., & Bajec, M. (2026). Obširna evalvacija komercialnih velikih jezikovnih modelov na področju sklepanja v slovenskem jeziku in slovnice. Uporabna informatika, 34(1). https://doi.org/10.31449/upinf.270

Vancouver

Malenšek M, Vreš D, Bajec M. Obširna evalvacija komercialnih velikih jezikovnih modelov na področju sklepanja v slovenskem jeziku in slovnice. Uporabna informatika. 2026;34(1). doi:10.31449/upinf.270

BibTeX
@article{Malenek2026,
  author = {Malenšek, Miha and Vreš, Domen and Bajec, Marko},
  title = {Obširna evalvacija komercialnih velikih jezikovnih modelov na področju sklepanja v slovenskem jeziku in slovnice},
  journal = {Uporabna informatika},
  year = {2026},
  volume = {34},
  number = {1},
  doi = {10.31449/upinf.270}
}
Kostenlos starten →