MultiLoKo: A Multilingual Local Knowledge Benchmark for LLMs Spanning 31 Languages
Computational Linguistics · 10.11.2026 · 1-33
DOI: 10.1162/coli.a.643 ↗Kurz gefasst
Fachartikel in der Zeitschrift Computational Linguistics, online erschienen am 10.11.2026. Verfasst von Dieuwke Hupkes und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
We present MultiLoKo, a new benchmark for evaluating multilinguality in LLMs covering 31 languages. MultiLoKo consists of three partitions: a main partition with 500 questions per language, separately sourced to be locally relevant to the specific language, and two translated partitions, containing human-authored translations from 30 non-English languages to English and vice versa. For comparison, we also release corresponding machine-authored translations. The data is equally distributed over two splits: a dev split and a blind, out-of-distribution test split. MultiLoKo can be used to study a variety of questions regarding the multilinguality of LLMs as well as meta-questions about multilingual benchmark creation. We compute MultiLoKo scores for 11 base and chat models marketed to be multilingual and study their average performance, their performance parity across languages, how much their ability to answer questions depends on the question language, and which languages are most difficult. We find that none of the models we evaluated perform well on MultiLoKo, as indicated by low average scores as well as large differences between the best and worst scoring languages. Furthermore, we find a substantial effect of the question language, indicating suboptimal knowledge transfer between languages. Lastly, we find that using local vs. English-translated data can result in differences of more than 20 points for the best performing models, drastically changing the estimated difficulty of some languages. For using machines instead of human translations, we find a weaker effect on ordering of language difficulty, a larger difference in model rankings, and a substantial drop in estimated performance for all models.
Zitieren
APA
Hupkes, D., & Bogoychev, N. (2026). MultiLoKo: A Multilingual Local Knowledge Benchmark for LLMs Spanning 31 Languages. Computational Linguistics, 1-33. https://doi.org/10.1162/coli.a.643
Vancouver
Hupkes D, Bogoychev N. MultiLoKo: A Multilingual Local Knowledge Benchmark for LLMs Spanning 31 Languages. Computational Linguistics. 2026:1-33. doi:10.1162/coli.a.643
BibTeX
@article{Hupkes2026,
author = {Hupkes, Dieuwke and Bogoychev, Nikolay},
title = {MultiLoKo: A Multilingual Local Knowledge Benchmark for LLMs Spanning 31 Languages},
journal = {Computational Linguistics},
year = {2026},
pages = {1--33},
doi = {10.1162/coli.a.643}
}
EUVIMED