EUVIMEDEUVIMED entdecken
Crossref

LocalBench: Benchmarking LLMs on County-Level Local Knowledge and Reasoning

Zihan Gao, Yifei Xu, Jacob Thebault-Spieker

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (45) · 38487-38495

DOI: 10.1609/aaai.v40i45.41190 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Zihan Gao und 2 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) have been widely evaluated on macro-scale geographic tasks, such as global factual recall, event summarization, and regional reasoning. Yet, their ability to handle hyper-local knowledge remains poorly understood. This gap is increasingly consequential as real-world applications, from civic platforms to community journalism, demand AI systems that can reason about neighborhood-specific dynamics, cultural narratives, and local governance. Existing benchmarks fall short in capturing this complexity, often relying on coarse-grained data or isolated references. We present LocalBench, the first benchmark designed to systematically evaluate LLMs on county-level local knowledge across the United States. Grounded in the Localness Conceptual Framework, LocalBench includes 14,782 validated question-answer pairs across 526 U.S. counties in 49 states, integrating diverse sources such as Census statistics, local subreddit discourse, and regional news. It spans physical, cognitive, and relational dimensions of locality. Using LocalBench, we evaluate 13 state-of-the-art LLMs under both closed-book and web-augmented settings. Our findings reveal critical limitations: even the best-performing models reach only 56.8% accuracy on narrative-style questions and perform below 15.5% on numerical reasoning. Moreover, larger model size and web augmentation do not guarantee better performance, for example, search improves Gemini's accuracy by +13.6%, but reduces GPT-series performance by -11.4%. These results underscore the urgent need for language models that can support equitable, place-aware AI systems: capable of engaging with the diverse, fine-grained realities of local communities across geographic and cultural contexts.

Zitieren

APA

Gao, Z., Xu, Y., & Thebault-Spieker, J. (2026). LocalBench: Benchmarking LLMs on County-Level Local Knowledge and Reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 40(45), 38487-38495. https://doi.org/10.1609/aaai.v40i45.41190

Vancouver

Gao Z, Xu Y, Thebault-Spieker J. LocalBench: Benchmarking LLMs on County-Level Local Knowledge and Reasoning. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(45):38487-38495. doi:10.1609/aaai.v40i45.41190

BibTeX
@article{Gao2026,
  author = {Gao, Zihan and Xu, Yifei and Thebault-Spieker, Jacob},
  title = {LocalBench: Benchmarking LLMs on County-Level Local Knowledge and Reasoning},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {45},
  pages = {38487--38495},
  doi = {10.1609/aaai.v40i45.41190}
}
Kostenlos starten →