EUVIMEDEUVIMED entdecken
Crossref

LexInstructEval: Lexical Instruction Following Evaluation for Large Language Models

Huimin Ren, Yan Liang, Baiqiao Su, Chaobo Sun, Hengtong Lu, Kaike Zhang, Chen Wei

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (30) · 25117-25123

DOI: 10.1609/aaai.v40i30.39701 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Huimin Ren und 6 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The ability of Large Language Models (LLMs) to precisely follow complex and fine-grained lexical instructions is a cornerstone of their utility and controllability. However, evaluating this capability remains a significant challenge. Current methods either rely on subjective and costly human evaluation or on automated ``LLM-as-a-judge'' systems, which suffer from inherent biases and unreliability. Existing programmatic benchmarks, while objective, often lack the expressiveness to test intricate, compositional constraints at a granular level. To address these limitations, we introduce LexInstructEval, a new benchmark and evaluation framework for fine-grained lexical instruction following. Our framework is built upon a formal, rule-based grammar that deconstructs complex instructions into a canonical (Procedure, Relation, Value) triplet. This grammar enables the systematic generation of a diverse dataset through a multi-stage, human-in-the-loop pipeline and facilitates objective verification via a transparent, programmatic engine. We release our dataset and open-source evaluation tools to facilitate further research into the controllability and reliability of LLMs.

Zitieren

APA

Ren, H., Liang, Y., Su, B., Sun, C., Lu, H., Zhang, K., & Wei, C. (2026). LexInstructEval: Lexical Instruction Following Evaluation for Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 40(30), 25117-25123. https://doi.org/10.1609/aaai.v40i30.39701

Vancouver

Ren H, Liang Y, Su B, Sun C, Lu H, Zhang K, et al. LexInstructEval: Lexical Instruction Following Evaluation for Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(30):25117-25123. doi:10.1609/aaai.v40i30.39701

BibTeX
@article{Ren2026,
  author = {Ren, Huimin and Liang, Yan and Su, Baiqiao and Sun, Chaobo and Lu, Hengtong and Zhang, Kaike and Wei, Chen},
  title = {LexInstructEval: Lexical Instruction Following Evaluation for Large Language Models},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {30},
  pages = {25117--25123},
  doi = {10.1609/aaai.v40i30.39701}
}
Kostenlos starten →