Evaluating LLMs for Police Decision-Making: A Framework Based on Police Action Scenarios
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (45) · 38817-38825
DOI: 10.1609/aaai.v40i45.41226 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Sangyub Lee und 2 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref).
Abstract (Originalsprache)
The use of Large Language Models (LLMs) in police opera- tions is growing, yet an evaluation framework tailored to po- lice operations remains absent. While LLM’s responses may not always be legally “incorrect”, their unverified use still can lead to severe issues such as unlawful arrests and improper evidence collection. To address this, we propose PAS (Po- lice Action Scenarios), a systematic framework covering the entire evaluation process. Applying this framework, we con- structed a novel QA dataset from over 8,000 official docu- ments and established key metrics validated through statis- tical analysis with police expert judgements. Experimental results show that commercial LLMs struggle with our new police-related tasks, particularly in providing fact-based rec- ommendations. This study highlights the necessity of an ex- pandable evaluation framework to ensure reliable AI-driven police operations. We release our data and prompt template.
Zitieren
APA
Lee, S., Kim, H., & Kim, H. (2026). Evaluating LLMs for Police Decision-Making: A Framework Based on Police Action Scenarios. Proceedings of the AAAI Conference on Artificial Intelligence, 40(45), 38817-38825. https://doi.org/10.1609/aaai.v40i45.41226
Vancouver
Lee S, Kim H, Kim H. Evaluating LLMs for Police Decision-Making: A Framework Based on Police Action Scenarios. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(45):38817-38825. doi:10.1609/aaai.v40i45.41226
BibTeX
@article{Lee2026,
author = {Lee, Sangyub and Kim, Heedou and Kim, Hyeoncheol},
title = {Evaluating LLMs for Police Decision-Making: A Framework Based on Police Action Scenarios},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {45},
pages = {38817--38825},
doi = {10.1609/aaai.v40i45.41226}
}
EUVIMED