Extracting Sentence Embeddings from Pretrained Transformer Models
Applied Sciences · 02.10.2024 · 14 (19) · 8887
DOI: 10.3390/app14198887 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 02.10.2024. Verfasst von Lukas Stankevičius und 1 weiteren Autor:innen. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 232 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Pre-trained transformer models shine in many natural language processing tasks and therefore are expected to bear the representation of the input sentence or text meaning. These sentence-level embeddings are also important in retrieval-augmented generation. But do commonly used plain averaging or prompt templates sufficiently capture and represent the underlying meaning? After providing a comprehensive review of existing sentence embedding extraction and refinement methods, we thoroughly test different combinations and our original extensions of the most promising ones on pretrained models. Namely, given 110 M parameters, BERT’s hidden representations from multiple layers, and many tokens, we try diverse ways to extract optimal sentence embeddings. We test various token aggregation and representation post-processing techniques. We also test multiple ways of using a general Wikitext dataset to complement BERT’s sentence embeddings. All methods are tested on eight Semantic Textual Similarity (STS), six short text clustering, and twelve classification tasks. We also evaluate our representation-shaping techniques on other static models, including random token representations. Proposed representation extraction methods improve the performance on STS and clustering tasks for all models considered. Very high improvements for static token-based models, especially random embeddings for STS tasks, almost reach the performance of BERT-derived representations. Our work shows that the representation-shaping techniques significantly improve sentence embeddings extracted from BERT-based and simple baseline models.
Zitieren
APA
Stankevičius, L., & Lukoševičius, M. (2024). Extracting Sentence Embeddings from Pretrained Transformer Models. Applied Sciences, 14(19), 8887. https://doi.org/10.3390/app14198887
Vancouver
Stankevičius L, Lukoševičius M. Extracting Sentence Embeddings from Pretrained Transformer Models. Applied Sciences. 2024;14(19):8887. doi:10.3390/app14198887
BibTeX
@article{Stankeviius2024,
author = {Stankevičius, Lukas and Lukoševičius, Mantas},
title = {Extracting Sentence Embeddings from Pretrained Transformer Models},
journal = {Applied Sciences},
year = {2024},
volume = {14},
number = {19},
pages = {8887},
doi = {10.3390/app14198887}
}
EUVIMED