ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (37) · 30969-30977
DOI: 10.1609/aaai.v40i37.40356 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Xingwei He und 6 weiteren Autor:innen.
Abstract (Originalsprache)
Instruction-following is a critical capability of Large Language Models (LLMs). While existing works primarily focus on assessing how well LLMs adhere to user instructions, they often overlook scenarios where instructions contain conflicting constraints—a common occurrence in complex prompts. The behavior of LLMs under such conditions remains under-explored. To bridge this gap, we introduce ConInstruct, a benchmark specifically designed to assess LLMs' ability to detect and resolve conflicts within user instructions. Using this dataset, we evaluate LLMs' conflict detection performance and analyze their conflict resolution behavior. Our experiments reveal two key findings: (1) Most proprietary LLMs exhibit strong conflict detection capabilities, whereas among open-source models, only DeepSeek-R1 demonstrates similarly strong performance. DeepSeek-R1 and Claude-4.5-Sonnet achieve the highest average F1-scores at 91.5% and 87.3%, respectively, ranking first and second overall. (2) Despite their strong conflict detection abilities, LLMs rarely explicitly notify users about the conflicts or request clarification when faced with conflicting constraints. These results underscore a critical shortcoming in current LLMs and highlight an important area for future improvement when designing instruction-following LLMs.
Zitieren
APA
He, X., Zhang, Q., Chen, P., Chen, G., Yu, L., Yuan, Y., & Yiu, S. (2026). ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions. Proceedings of the AAAI Conference on Artificial Intelligence, 40(37), 30969-30977. https://doi.org/10.1609/aaai.v40i37.40356
Vancouver
He X, Zhang Q, Chen P, Chen G, Yu L, Yuan Y, et al. ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(37):30969-30977. doi:10.1609/aaai.v40i37.40356
BibTeX
@article{He2026,
author = {He, Xingwei and Zhang, Qianru and Chen, Pengfei and Chen, Guanhua and Yu, Linlin and Yuan, Yuan and Yiu, Siu-Ming},
title = {ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {37},
pages = {30969--30977},
doi = {10.1609/aaai.v40i37.40356}
}
EUVIMED