Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls
ACM Transactions on Information Systems · 31.07.2023 · 41 (3) · 1-40
DOI: 10.1145/3570724 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Information Systems, online erschienen am 10.04.2023. Verfasst von Hang Li und 4 weiteren Autor:innen. Die Arbeit wurde bisher 23-mal zitiert (Crossref) und nennt 83 Literaturangaben. Gefördert von: Grain Research and Development Corporation project AgAsk, Australian Research Council DECRA Research Fellowship.
Abstract (Originalsprache)
Pseudo Relevance Feedback (PRF) is known to improve the effectiveness of bag-of-words retrievers. At the same time, deep language models have been shown to outperform traditional bag-of-words rerankers. However, it is unclear how to integrate PRF directly with emergent deep language models. This article addresses this gap by investigating methods for integrating PRF signals with rerankers and dense retrievers based on deep language models. We consider text-based, vector-based and hybrid PRF approaches and investigate different ways of combining and scoring relevance signals. An extensive empirical evaluation was conducted across four different datasets and two task settings (retrieval and ranking). Text-based PRF results show that the use of PRF had a mixed effect on deep rerankers across different datasets. We found that the best effectiveness was achieved when (i) directly concatenating each PRF passage with the query, searching with the new set of queries, and then aggregating the scores; (ii) using Borda to aggregate scores from PRF runs. Vector-based PRF results show that the use of PRF enhanced the effectiveness of deep rerankers and dense retrievers over several evaluation metrics. We found that higher effectiveness was achieved when (i) the query retains either the majority or the same weight within the PRF mechanism, and (ii) a shallower PRF signal (i.e., a smaller number of top-ranked passages) was employed, rather than a deeper signal. Our vector-based PRF method is computationally efficient; thus, this represents a general PRF method others can use with deep rerankers and dense retrievers.
Förderung
- Grain Research and Development Corporation project AgAsk
- Australian Research Council DECRA Research Fellowship
Zitieren
APA
Li, H., Mourad, A., Zhuang, S., Koopman, B., & Zuccon, G. (2023). Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls. ACM Transactions on Information Systems, 41(3), 1-40. https://doi.org/10.1145/3570724
Vancouver
Li H, Mourad A, Zhuang S, Koopman B, Zuccon G. Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls. ACM Transactions on Information Systems. 2023;41(3):1-40. doi:10.1145/3570724
BibTeX
@article{Li2023,
author = {Li, Hang and Mourad, Ahmed and Zhuang, Shengyao and Koopman, Bevan and Zuccon, Guido},
title = {Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls},
journal = {ACM Transactions on Information Systems},
year = {2023},
volume = {41},
number = {3},
pages = {1--40},
doi = {10.1145/3570724}
}
EUVIMED