Calibration and context in human evaluation of machine translation
Natural Language Processing · 2025-07 · 31 (4) · 1017-1041
DOI: 10.1017/nlp.2024.5 ↗Kurz gefasst
Fachartikel in der Zeitschrift Natural Language Processing, online erschienen am 03.06.2024. Verfasst von Rebecca Knowles und 1 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Human evaluation of machine translation is considered the “gold standard” for evaluation, but it remains a challenging task for which to define best practices. Recent work has focused on incorporating intersentential context into human evaluation, to better distinguish between high-performing machine translation systems and human translations. In this work, we examine several ways that such context influences evaluation and evaluation protocols. We take a close look at annotator variation through the lens of calibration sets and focus on the implications for context-aware evaluation protocols. We then demonstrate one way in which degraded target-side intersentential context can influence annotator scores of individual sentences, a finding that supports the context-aware approach to evaluation and which also has implications for best practices in evaluation protocols.
Zitieren
APA
Knowles, R., & Lo, C. (2025). Calibration and context in human evaluation of machine translation. Natural Language Processing, 31(4), 1017-1041. https://doi.org/10.1017/nlp.2024.5
Vancouver
Knowles R, Lo C. Calibration and context in human evaluation of machine translation. Natural Language Processing. 2025;31(4):1017-1041. doi:10.1017/nlp.2024.5
BibTeX
@article{Knowles2025,
author = {Knowles, Rebecca and Lo, Chi-kiu},
title = {Calibration and context in human evaluation of machine translation},
journal = {Natural Language Processing},
year = {2025},
volume = {31},
number = {4},
pages = {1017--1041},
doi = {10.1017/nlp.2024.5}
}
EUVIMED