EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Exploring Practical Gaps in Using Cross Entropy to Implement Maximum Mutual Information Criterion for Rationalization

Wei Liu, Zhiying Deng, Zhongyu Niu, Jun Wang, Haozhao Wang, Ruixuan Li

Transactions of the Association for Computational Linguistics · 27.06.2025 · 13 · 577-594

DOI: 10.1162/tacl_a_00758 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 27.06.2025. Verfasst von Wei Liu und 5 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Rationalization is a framework that aims to build self-explanatory NLP models by extracting a subset of human-intelligible pieces of their inputting texts. It involves a cooperative game where a selector selects the most human-intelligible parts of the input as the rationale, followed by a predictor that makes predictions based on these selected rationales. Existing literature uses the cross-entropy between the model’s predictions and the ground-truth labels to measure the informativeness of the selected rationales, guiding the selector to choose better ones. In this study, we first theoretically analyze the objective of rationalization by decomposing it into two parts: the model-agnostic informativeness of the rationale candidates and the predictor’s degree of fit. We then provide various empirical evidence to support that, under this framework, the selector tends to sample from a limited small region, causing the predictor to overfit these localized areas. This results in a significant mismatch between the cross-entropy objective and the informativeness of the rationale candidates, leading to suboptimal solutions. To address this issue, we propose a simple yet effective method that introduces random vicinal1 perturbations to the selected rationale candidates. This approach broadens the predictor’s assessment to a vicinity around the selected rationale candidate. Compared to recent competitive methods, our method significantly improves rationale quality (by up to 6.6%) across six widely used classification datasets. The term “vicinal” is borrowed from vicinal risk minimization (Chapelle et al., 2000); “vicinal” means neighboring or adjacent.

Zitieren

APA

Liu, W., Deng, Z., Niu, Z., Wang, J., Wang, H., & Li, R. (2025). Exploring Practical Gaps in Using Cross Entropy to Implement Maximum Mutual Information Criterion for Rationalization. Transactions of the Association for Computational Linguistics, 13, 577-594. https://doi.org/10.1162/tacl_a_00758

Vancouver

Liu W, Deng Z, Niu Z, Wang J, Wang H, Li R. Exploring Practical Gaps in Using Cross Entropy to Implement Maximum Mutual Information Criterion for Rationalization. Transactions of the Association for Computational Linguistics. 2025;13:577-594. doi:10.1162/tacl_a_00758

BibTeX
@article{Liu2025,
  author = {Liu, Wei and Deng, Zhiying and Niu, Zhongyu and Wang, Jun and Wang, Haozhao and Li, Ruixuan},
  title = {Exploring Practical Gaps in Using Cross Entropy to Implement Maximum Mutual Information Criterion for Rationalization},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2025},
  volume = {13},
  pages = {577--594},
  doi = {10.1162/tacl_a_00758}
}
Kostenlos starten →