EUVIMEDEUVIMED entdecken
Crossref

Evaluating LLMs for Police Decision-Making: A Framework Based on Police Action Scenarios

Sangyub Lee, Heedou Kim, Hyeoncheol Kim

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (45) · 38817-38825

DOI: 10.1609/aaai.v40i45.41226 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Sangyub Lee und 2 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The use of Large Language Models (LLMs) in police opera- tions is growing, yet an evaluation framework tailored to po- lice operations remains absent. While LLM’s responses may not always be legally “incorrect”, their unverified use still can lead to severe issues such as unlawful arrests and improper evidence collection. To address this, we propose PAS (Po- lice Action Scenarios), a systematic framework covering the entire evaluation process. Applying this framework, we con- structed a novel QA dataset from over 8,000 official docu- ments and established key metrics validated through statis- tical analysis with police expert judgements. Experimental results show that commercial LLMs struggle with our new police-related tasks, particularly in providing fact-based rec- ommendations. This study highlights the necessity of an ex- pandable evaluation framework to ensure reliable AI-driven police operations. We release our data and prompt template.

Zitieren

APA

Lee, S., Kim, H., & Kim, H. (2026). Evaluating LLMs for Police Decision-Making: A Framework Based on Police Action Scenarios. Proceedings of the AAAI Conference on Artificial Intelligence, 40(45), 38817-38825. https://doi.org/10.1609/aaai.v40i45.41226

Vancouver

Lee S, Kim H, Kim H. Evaluating LLMs for Police Decision-Making: A Framework Based on Police Action Scenarios. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(45):38817-38825. doi:10.1609/aaai.v40i45.41226

BibTeX
@article{Lee2026,
  author = {Lee, Sangyub and Kim, Heedou and Kim, Hyeoncheol},
  title = {Evaluating LLMs for Police Decision-Making: A Framework Based on Police Action Scenarios},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {45},
  pages = {38817--38825},
  doi = {10.1609/aaai.v40i45.41226}
}
Kostenlos starten →