EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Rethinking Evaluation in Simultaneous Speech Translation: A Case for Monotonic Test Sets

Mana Makinae, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Computational Linguistics · 11.11.2026 · 1-30

DOI: 10.1162/coli.a.622 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computational Linguistics, online erschienen am 11.11.2026. Verfasst von Mana Makinae und 3 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Overcoming the trade-off between quality and latency is a challenge in simultaneous speech translation. Common approaches in previous work have been to segment the source sentence or align target sentences with the source’s syntax as closely as possible, enabling faster translations while maintaining quality. However, a major limitation in these studies is the reliance on existing translation test data, which often includes reordering and are unsuitable for simultaneous settings with low latency. Alternatively, some use interpretation data transcribed from interpreters, which is also problematic due to translation errors and omissions, making both inadequate for fully evaluating simultaneous models. In this work, we introduce a construction, verification, and analysis of a new test set specifically designed for simultaneous settings, with a focus on maintaining word and phrase order consistency with the source. The test set comprises three language pairs representing different levels of word order similarity to the source by leveraging large language models, with quality verified by professional interpreters. This provides an interpreter-grounded perspective, and empirically shows the ideal level of monotonicity and other sentence style characteristics including syntax simplicity and sentence length. It also reveals the capabilities and limitations of LLMs on monotonic translation. Experiments revealed that existing test data tends to underestimate a model’s performance, while the proposed test set, simul-tst-COMMON, offers a more appropriate evaluation of simultaneous models. Moreover, the quality gap between wait-k and Local Agreement suggests that the adaptive policy more closely resembles the monotonic translation behavior of human interpreters. Finally, the analysis highlights the limitations of current metrics, which may not be fully suitable for evaluating simultaneous tasks.

Zitieren

APA

Makinae, M., Sakai, Y., Kamigaito, H., & Watanabe, T. (2026). Rethinking Evaluation in Simultaneous Speech Translation: A Case for Monotonic Test Sets. Computational Linguistics, 1-30. https://doi.org/10.1162/coli.a.622

Vancouver

Makinae M, Sakai Y, Kamigaito H, Watanabe T. Rethinking Evaluation in Simultaneous Speech Translation: A Case for Monotonic Test Sets. Computational Linguistics. 2026:1-30. doi:10.1162/coli.a.622

BibTeX
@article{Makinae2026,
  author = {Makinae, Mana and Sakai, Yusuke and Kamigaito, Hidetaka and Watanabe, Taro},
  title = {Rethinking Evaluation in Simultaneous Speech Translation: A Case for Monotonic Test Sets},
  journal = {Computational Linguistics},
  year = {2026},
  pages = {1--30},
  doi = {10.1162/coli.a.622}
}
Kostenlos starten →