From Robustness to Improved Generalization and Calibration in Pre-trained Language Models
Transactions of the Association for Computational Linguistics · 19.03.2025 · 13 · 264-280
DOI: 10.1162/tacl_a_00739 ↗Kurz gefasst
Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 19.03.2025. Verfasst von Josip Jukić und 1 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 58 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Enforcing representation smoothness in pre-trained language models (PLMs) through Jacobian and Hessian regularization provides an effective approach for enhancing both robustness and generalization. Although such regularization methods have proven effective in computer vision, their application in natural language processing, where PLM inputs are derived from a discrete domain, poses unique challenges. We introduce JacHess, a regularization approach for PLMs that minimizes the norms of the Jacobian and Hessian matrices in intermediate representations, using embeddings as substitutes for discrete token inputs. JacHess supports dual-mode regularization, alternating between fine-tuning with labeled data and regularization with unlabeled data. We evaluate JacHess on the GLUE benchmark and demonstrate that it consistently and significantly improves in-distribution generalization and enhances performance under domain shift. Across diverse PLMs, JacHess outperforms comparable representation-based regularization methods and unregularized fine-tuning, while also improving model calibration. Our findings, coupled with a computationally efficient estimator for the Jacobian and Hessian norms, position JacHess as a robust and widely applicable solution for enhancing PLM performance.
Zitieren
APA
Jukić, J., & Šnajder, J. (2025). From Robustness to Improved Generalization and Calibration in Pre-trained Language Models. Transactions of the Association for Computational Linguistics, 13, 264-280. https://doi.org/10.1162/tacl_a_00739
Vancouver
Jukić J, Šnajder J. From Robustness to Improved Generalization and Calibration in Pre-trained Language Models. Transactions of the Association for Computational Linguistics. 2025;13:264-280. doi:10.1162/tacl_a_00739
BibTeX
@article{Juki2025,
author = {Jukić, Josip and Šnajder, Jan},
title = {From Robustness to Improved Generalization and Calibration in Pre-trained Language Models},
journal = {Transactions of the Association for Computational Linguistics},
year = {2025},
volume = {13},
pages = {264--280},
doi = {10.1162/tacl_a_00739}
}
EUVIMED