Beyond Cultural Trivia: Calibrated Exemplar Retrieval, Mode-Aware Routing, and Selective Abstention for Cross-Cultural LLM Services
Stout in Humanities and Social Sciences · 20.01.2026 · 2 (1) · 24-42
DOI: 10.61424/33995e23 ↗Kurz gefasst
Fachartikel in der Zeitschrift Stout in Humanities and Social Sciences, online erschienen am 20.01.2026. Verfasst von Daisy Zhu.
Abstract (Originalsprache)
Cross-cultural language-model services need more than high average accuracy: they need inference procedures that recognize ambiguous answer structures, communicate uncertainty, and avoid concentrating errors in underrepresented regions. This study evaluates such procedures on the 2024 public release of CulturalBench, comprising 1,227 multiple-choice questions and 4,908 option-level binary judgments from 45 countries or regions. A deterministic decision-layer test bed was used to isolate retrieval, score verification, calibration, and abstention from changes in proprietary model checkpoints. Word- and character-ngram scorers, global and country exemplar retrieval, score fusion, a score-stack verifier, an observable mode-aware router, temperature scaling, and confidence-based selective prediction were evaluated with country-stratified, question-grouped five-fold cross-validation. The text-only scorer achieved 40.51% accuracy on CulturalBench-Easy and 9.70% question-level exact match on CulturalBench-Hard. A position-only shortcut reached 53.63% and 51.75%, exposing substantial answer-position regularity. The mode-aware router increased these scores to 54.69% and 52.32%, respectively; temperature scaling preserved the predictions while reducing expected calibration error to 0.0234 on Easy and 0.0131 on Hard. At 80% coverage, the calibrated router achieved 56.62% Easy accuracy and 54.79% Hard exact match. Nevertheless, calibrated regional scores ranged from 44.09% to 70.37% on Easy and from 40.16% to 66.67% on Hard. Temperature scaling improved calibration, and abstention lowered risk at intermediate coverage, but neither high confidence nor aggregate performance established cross-cultural robustness. Position balance, multi-answer sensitivity, worst-region reporting, and group-aware coverage should therefore be treated as core requirements for evaluating cross-cultural language-model services.
Zitieren
APA
Zhu, D. (2026). Beyond Cultural Trivia: Calibrated Exemplar Retrieval, Mode-Aware Routing, and Selective Abstention for Cross-Cultural LLM Services. Stout in Humanities and Social Sciences, 2(1), 24-42. https://doi.org/10.61424/33995e23
Vancouver
Zhu D. Beyond Cultural Trivia: Calibrated Exemplar Retrieval, Mode-Aware Routing, and Selective Abstention for Cross-Cultural LLM Services. Stout in Humanities and Social Sciences. 2026;2(1):24-42. doi:10.61424/33995e23
BibTeX
@article{Zhu2026,
author = {Zhu, Daisy},
title = {Beyond Cultural Trivia: Calibrated Exemplar Retrieval, Mode-Aware Routing, and Selective Abstention for Cross-Cultural LLM Services},
journal = {Stout in Humanities and Social Sciences},
year = {2026},
volume = {2},
number = {1},
pages = {24--42},
doi = {10.61424/33995e23}
}
EUVIMED