EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval

Sheng-Chieh Lin, Minghan Li, Jimmy Lin

Transactions of the Association for Computational Linguistics · 18.05.2023 · 11 · 436-452

DOI: 10.1162/tacl_a_00556 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 18.05.2023. Verfasst von Sheng-Chieh Lin und 2 weiteren Autor:innen. Die Arbeit wurde bisher 24-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Pre-trained language models have been successful in many knowledge-intensive NLP tasks. However, recent work has shown that models such as BERT are not “structurally ready” to aggregate textual information into a [CLS] vector for dense passage retrieval (DPR). This “lack of readiness” results from the gap between language model pre-training and DPR fine-tuning. Previous solutions call for computationally expensive techniques such as hard negative mining, cross-encoder distillation, and further pre-training to learn a robust DPR model. In this work, we instead propose to fully exploit knowledge in a pre-trained language model for DPR by aggregating the contextualized token embeddings into a dense vector, which we call agg★. By concatenating vectors from the [CLS] token and agg★, our Aggretriever model substantially improves the effectiveness of dense retrieval models on both in-domain and zero-shot evaluations without introducing substantial training overhead. Code is available at https://github.com/castorini/dhr.

Zitieren

APA

Lin, S., Li, M., & Lin, J. (2023). Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval. Transactions of the Association for Computational Linguistics, 11, 436-452. https://doi.org/10.1162/tacl_a_00556

Vancouver

Lin S, Li M, Lin J. Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval. Transactions of the Association for Computational Linguistics. 2023;11:436-452. doi:10.1162/tacl_a_00556

BibTeX
@article{Lin2023,
  author = {Lin, Sheng-Chieh and Li, Minghan and Lin, Jimmy},
  title = {Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  volume = {11},
  pages = {436--452},
  doi = {10.1162/tacl_a_00556}
}
Kostenlos starten →