Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries
Transactions of the Association for Computational Linguistics · 29.07.2025 · 13 · 878-900
DOI: 10.1162/tacl.a.24 ↗Kurz gefasst
Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 29.07.2025. Verfasst von Chaitanya Malaviya und 5 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Language model users often issue queries that lack specification, where the context under which a query was issued—such as the user’s identity, the query’s intent, and the criteria for a response to be useful—is not explicit. For instance, a good response to a subjective query like “What book should I read next?” would depend on the user’s preferences, and a good response to an open-ended query like “How do antibiotics work against bacteria?” would depend on the user’s expertise. This makes evaluation of responses to such queries an ill-posed task, as evaluators may make arbitrary judgments about the response quality. To remedy this, we present contextualized evaluations, a protocol that synthetically constructs context surrounding an underspecified query and provides it during evaluation. We find that the presence of context can 1) alter conclusions drawn from evaluation, even flipping benchmark rankings between model pairs, 2) nudge evaluators to make fewer judgments based on surface-level criteria, like style, and 3) provide new insights about model behavior across diverse contexts. Specifically, our procedure suggests a potential bias towards WEIRD (Western, Educated, Industrialized, Rich and Democratic) contexts in models’ “default” responses and we find that models are not equally sensitive to following different contexts, even when they are provided in prompts.1
Zitieren
APA
Malaviya, C., Chang, J. C., Roth, D., Iyyer, M., Yatskar, M., & Lo, K. (2025). Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries. Transactions of the Association for Computational Linguistics, 13, 878-900. https://doi.org/10.1162/tacl.a.24
Vancouver
Malaviya C, Chang JC, Roth D, Iyyer M, Yatskar M, Lo K. Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries. Transactions of the Association for Computational Linguistics. 2025;13:878-900. doi:10.1162/tacl.a.24
BibTeX
@article{Malaviya2025,
author = {Malaviya, Chaitanya and Chang, Joseph Chee and Roth, Dan and Iyyer, Mohit and Yatskar, Mark and Lo, Kyle},
title = {Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries},
journal = {Transactions of the Association for Computational Linguistics},
year = {2025},
volume = {13},
pages = {878--900},
doi = {10.1162/tacl.a.24}
}
EUVIMED