EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Dataset

Jindrich Libovický, Jindrich Helcl, Andrei-Alexandru Manea, Gianluca Vico

Transactions of the Association for Computational Linguistics · 01.07.2026 · 14 · 1461-1488

DOI: 10.1162/tacl.a.737 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 01.07.2026. Verfasst von Jindrich Libovický und 3 weiteren Autor:innen. Die Arbeit nennt 55 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We introduce CUS-QA, a benchmark for evaluation of open-ended regional question answering that encompasses both textual and visual modalities. We also provide strong baselines using state-of-the-art large language models (LLMs). Our dataset consists of manually curated questions and answers grounded in Wikipedia, created by native speakers from Czechia, Slovakia, and Ukraine, with accompanying English translations. It includes both purely textual questions and those requiring visual understanding. We evaluate state-of-the-art LLMs through prompting and add human judgments of answer correctness. Using these human evaluations, we analyze the reliability of existing automatic evaluation metrics. Our baseline results show that even the best open-weight LLMs achieve only over 40% accuracy on textual questions and below 30% on visual questions. LLM-based evaluation metrics show strong correlation with human judgment, while traditional string-overlap metrics perform surprisingly well due to the prevalence of named entities in answers.

Zitieren

APA

Libovický, J., Helcl, J., Manea, A., & Vico, G. (2026). CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Dataset. Transactions of the Association for Computational Linguistics, 14, 1461-1488. https://doi.org/10.1162/tacl.a.737

Vancouver

Libovický J, Helcl J, Manea A, Vico G. CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Dataset. Transactions of the Association for Computational Linguistics. 2026;14:1461-1488. doi:10.1162/tacl.a.737

BibTeX
@article{Libovick2026,
  author = {Libovický, Jindrich and Helcl, Jindrich and Manea, Andrei-Alexandru and Vico, Gianluca},
  title = {CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Dataset},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  volume = {14},
  pages = {1461--1488},
  doi = {10.1162/tacl.a.737}
}
Kostenlos starten →