Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval
Transactions of the Association for Computational Linguistics · 18.05.2023 · 11 · 436-452
DOI: 10.1162/tacl_a_00556 ↗Kurz gefasst
Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 18.05.2023. Verfasst von Sheng-Chieh Lin und 2 weiteren Autor:innen. Die Arbeit wurde bisher 24-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Pre-trained language models have been successful in many knowledge-intensive NLP tasks. However, recent work has shown that models such as BERT are not “structurally ready” to aggregate textual information into a [CLS] vector for dense passage retrieval (DPR). This “lack of readiness” results from the gap between language model pre-training and DPR fine-tuning. Previous solutions call for computationally expensive techniques such as hard negative mining, cross-encoder distillation, and further pre-training to learn a robust DPR model. In this work, we instead propose to fully exploit knowledge in a pre-trained language model for DPR by aggregating the contextualized token embeddings into a dense vector, which we call agg★. By concatenating vectors from the [CLS] token and agg★, our Aggretriever model substantially improves the effectiveness of dense retrieval models on both in-domain and zero-shot evaluations without introducing substantial training overhead. Code is available at https://github.com/castorini/dhr.
Zitieren
APA
Lin, S., Li, M., & Lin, J. (2023). Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval. Transactions of the Association for Computational Linguistics, 11, 436-452. https://doi.org/10.1162/tacl_a_00556
Vancouver
Lin S, Li M, Lin J. Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval. Transactions of the Association for Computational Linguistics. 2023;11:436-452. doi:10.1162/tacl_a_00556
BibTeX
@article{Lin2023,
author = {Lin, Sheng-Chieh and Li, Minghan and Lin, Jimmy},
title = {Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval},
journal = {Transactions of the Association for Computational Linguistics},
year = {2023},
volume = {11},
pages = {436--452},
doi = {10.1162/tacl_a_00556}
}
EUVIMED