LexInstructEval: Lexical Instruction Following Evaluation for Large Language Models
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (30) · 25117-25123
DOI: 10.1609/aaai.v40i30.39701 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Huimin Ren und 6 weiteren Autor:innen.
Abstract (Originalsprache)
The ability of Large Language Models (LLMs) to precisely follow complex and fine-grained lexical instructions is a cornerstone of their utility and controllability. However, evaluating this capability remains a significant challenge. Current methods either rely on subjective and costly human evaluation or on automated ``LLM-as-a-judge'' systems, which suffer from inherent biases and unreliability. Existing programmatic benchmarks, while objective, often lack the expressiveness to test intricate, compositional constraints at a granular level. To address these limitations, we introduce LexInstructEval, a new benchmark and evaluation framework for fine-grained lexical instruction following. Our framework is built upon a formal, rule-based grammar that deconstructs complex instructions into a canonical (Procedure, Relation, Value) triplet. This grammar enables the systematic generation of a diverse dataset through a multi-stage, human-in-the-loop pipeline and facilitates objective verification via a transparent, programmatic engine. We release our dataset and open-source evaluation tools to facilitate further research into the controllability and reliability of LLMs.
Zitieren
APA
Ren, H., Liang, Y., Su, B., Sun, C., Lu, H., Zhang, K., & Wei, C. (2026). LexInstructEval: Lexical Instruction Following Evaluation for Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 40(30), 25117-25123. https://doi.org/10.1609/aaai.v40i30.39701
Vancouver
Ren H, Liang Y, Su B, Sun C, Lu H, Zhang K, et al. LexInstructEval: Lexical Instruction Following Evaluation for Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(30):25117-25123. doi:10.1609/aaai.v40i30.39701
BibTeX
@article{Ren2026,
author = {Ren, Huimin and Liang, Yan and Su, Baiqiao and Sun, Chaobo and Lu, Hengtong and Zhang, Kaike and Wei, Chen},
title = {LexInstructEval: Lexical Instruction Following Evaluation for Large Language Models},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {30},
pages = {25117--25123},
doi = {10.1609/aaai.v40i30.39701}
}
EUVIMED