EUVIMEDEUVIMED entdecken
Crossref

ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions

Xingwei He, Qianru Zhang, Pengfei Chen, Guanhua Chen, Linlin Yu, Yuan Yuan, Siu-Ming Yiu

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (37) · 30969-30977

DOI: 10.1609/aaai.v40i37.40356 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Xingwei He und 6 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Instruction-following is a critical capability of Large Language Models (LLMs). While existing works primarily focus on assessing how well LLMs adhere to user instructions, they often overlook scenarios where instructions contain conflicting constraints—a common occurrence in complex prompts. The behavior of LLMs under such conditions remains under-explored. To bridge this gap, we introduce ConInstruct, a benchmark specifically designed to assess LLMs' ability to detect and resolve conflicts within user instructions. Using this dataset, we evaluate LLMs' conflict detection performance and analyze their conflict resolution behavior. Our experiments reveal two key findings: (1) Most proprietary LLMs exhibit strong conflict detection capabilities, whereas among open-source models, only DeepSeek-R1 demonstrates similarly strong performance. DeepSeek-R1 and Claude-4.5-Sonnet achieve the highest average F1-scores at 91.5% and 87.3%, respectively, ranking first and second overall. (2) Despite their strong conflict detection abilities, LLMs rarely explicitly notify users about the conflicts or request clarification when faced with conflicting constraints. These results underscore a critical shortcoming in current LLMs and highlight an important area for future improvement when designing instruction-following LLMs.

Zitieren

APA

He, X., Zhang, Q., Chen, P., Chen, G., Yu, L., Yuan, Y., & Yiu, S. (2026). ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions. Proceedings of the AAAI Conference on Artificial Intelligence, 40(37), 30969-30977. https://doi.org/10.1609/aaai.v40i37.40356

Vancouver

He X, Zhang Q, Chen P, Chen G, Yu L, Yuan Y, et al. ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(37):30969-30977. doi:10.1609/aaai.v40i37.40356

BibTeX
@article{He2026,
  author = {He, Xingwei and Zhang, Qianru and Chen, Pengfei and Chen, Guanhua and Yu, Linlin and Yuan, Yuan and Yiu, Siu-Ming},
  title = {ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {37},
  pages = {30969--30977},
  doi = {10.1609/aaai.v40i37.40356}
}
Kostenlos starten →