EUVIMEDEUVIMED entdecken
Crossref

Rethinking the Evaluation Methods of LLMs' Grammatical Knowledge

Yusuke Ide, Yuto Nishida, Justin Vasselli, Miyu Oba, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Journal of Natural Language Processing · 2026 · 33 (1) · 51-75

DOI: 10.5715/jnlp.33.51 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Natural Language Processing, 2026. Verfasst von Yusuke Ide und 6 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The grammatical knowledge of language models (LMs) is often measured using a benchmark of linguistic minimal pairs, where the LMs are presented with a pair of acceptable and unacceptable sentences and required to judge which is more acceptable. Conventional approaches directly compare sentence probabilities assigned by LMs, but recent large language models (LLMs) are trained to perform tasks via prompting, and thus, the raw probabilities they assign may not fully reflect their grammatical knowledge. In this study, we attempt to derive more accurate acceptability judgments from LLMs using prompts and templates. Through extensive experiments in English and Chinese, we compare nine judgment methods and find two of them, a probability readout method—in-template LP and a prompt-based method—Yes/No probability computing, achieve higher accuracy than the conventional ones. We also find that ensembling the two methods outperforms single methods. Consequently, we recommend these techniques, either individually or ensembled, as more effective alternatives to conventional approaches for making the most of and comprehensively evaluating LLMs' grammatical knowledge.

Zitieren

APA

Ide, Y., Nishida, Y., Vasselli, J., Oba, M., Sakai, Y., Kamigaito, H., & Watanabe, T. (2026). Rethinking the Evaluation Methods of LLMs' Grammatical Knowledge. Journal of Natural Language Processing, 33(1), 51-75. https://doi.org/10.5715/jnlp.33.51

Vancouver

Ide Y, Nishida Y, Vasselli J, Oba M, Sakai Y, Kamigaito H, et al. Rethinking the Evaluation Methods of LLMs' Grammatical Knowledge. Journal of Natural Language Processing. 2026;33(1):51-75. doi:10.5715/jnlp.33.51

BibTeX
@article{Ide2026,
  author = {Ide, Yusuke and Nishida, Yuto and Vasselli, Justin and Oba, Miyu and Sakai, Yusuke and Kamigaito, Hidetaka and Watanabe, Taro},
  title = {Rethinking the Evaluation Methods of LLMs' Grammatical Knowledge},
  journal = {Journal of Natural Language Processing},
  year = {2026},
  volume = {33},
  number = {1},
  pages = {51--75},
  doi = {10.5715/jnlp.33.51}
}
Kostenlos starten →