EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

MENLI: Robust Evaluation Metrics from Natural Language Inference

Yanran Chen, Steffen Eger

Transactions of the Association for Computational Linguistics · 12.07.2023 · 11 · 804-825

DOI: 10.1162/tacl_a_00576 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 12.07.2023. Verfasst von Yanran Chen und 1 weiteren Autor:innen. Die Arbeit wurde bisher 28-mal zitiert (Crossref) und nennt 75 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recently proposed BERT-based evaluation metrics for text generation perform well on standard benchmarks but are vulnerable to adversarial attacks, e.g., relating to information correctness. We argue that this stems (in part) from the fact that they are models of semantic similarity. In contrast, we develop evaluation metrics based on Natural Language Inference (NLI), which we deem a more appropriate modeling. We design a preference-based adversarial attack framework and show that our NLI based metrics are much more robust to the attacks than the recent BERT-based metrics. On standard benchmarks, our NLI based metrics outperform existing summarization metrics, but perform below SOTA MT metrics. However, when combining existing metrics with our NLI metrics, we obtain both higher adversarial robustness (15%–30%) and higher quality metrics as measured on standard benchmarks (+5% to 30%).

Zitieren

APA

Chen, Y., & Eger, S. (2023). MENLI: Robust Evaluation Metrics from Natural Language Inference. Transactions of the Association for Computational Linguistics, 11, 804-825. https://doi.org/10.1162/tacl_a_00576

Vancouver

Chen Y, Eger S. MENLI: Robust Evaluation Metrics from Natural Language Inference. Transactions of the Association for Computational Linguistics. 2023;11:804-825. doi:10.1162/tacl_a_00576

BibTeX
@article{Chen2023,
  author = {Chen, Yanran and Eger, Steffen},
  title = {MENLI: Robust Evaluation Metrics from Natural Language Inference},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  volume = {11},
  pages = {804--825},
  doi = {10.1162/tacl_a_00576}
}
Kostenlos starten →