EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Calibration and context in human evaluation of machine translation

Rebecca Knowles, Chi-kiu Lo

Natural Language Processing · 2025-07 · 31 (4) · 1017-1041

DOI: 10.1017/nlp.2024.5 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Natural Language Processing, online erschienen am 03.06.2024. Verfasst von Rebecca Knowles und 1 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Human evaluation of machine translation is considered the “gold standard” for evaluation, but it remains a challenging task for which to define best practices. Recent work has focused on incorporating intersentential context into human evaluation, to better distinguish between high-performing machine translation systems and human translations. In this work, we examine several ways that such context influences evaluation and evaluation protocols. We take a close look at annotator variation through the lens of calibration sets and focus on the implications for context-aware evaluation protocols. We then demonstrate one way in which degraded target-side intersentential context can influence annotator scores of individual sentences, a finding that supports the context-aware approach to evaluation and which also has implications for best practices in evaluation protocols.

Zitieren

APA

Knowles, R., & Lo, C. (2025). Calibration and context in human evaluation of machine translation. Natural Language Processing, 31(4), 1017-1041. https://doi.org/10.1017/nlp.2024.5

Vancouver

Knowles R, Lo C. Calibration and context in human evaluation of machine translation. Natural Language Processing. 2025;31(4):1017-1041. doi:10.1017/nlp.2024.5

BibTeX
@article{Knowles2025,
  author = {Knowles, Rebecca and Lo, Chi-kiu},
  title = {Calibration and context in human evaluation of machine translation},
  journal = {Natural Language Processing},
  year = {2025},
  volume = {31},
  number = {4},
  pages = {1017--1041},
  doi = {10.1017/nlp.2024.5}
}
Kostenlos starten →