Revisiting Bag of Words Document Representations for Efficient Ranking with Transformers
ACM Transactions on Information Systems · 30.09.2024 · 42 (5) · 1-27
DOI: 10.1145/3640460 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Information Systems, online erschienen am 29.04.2024. Verfasst von David Rau und 2 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 60 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Netherlands Organization for Scientific Research, Facebook Research, Innovation Exchange Amsterdam.
Abstract (Originalsprache)
Modern transformer-based information retrieval models achieve state-of-the-art performance across various benchmarks. The self-attention of the transformer models is a powerful mechanism to contextualize terms over the whole input but quickly becomes prohibitively expensive for long input as required in document retrieval. Instead of focusing on the model itself to improve efficiency, this paper explores different bag of words document representations that encode full documents by only a fraction of their characteristic terms, allowing us to control and reduce the input length. We experiment with various models for document retrieval on MS MARCO data, as well as zero-shot document retrieval on Robust04, and show large gains in efficiency while retaining reasonable effectiveness. Inference time efficiency gains are both lowering the time and memory complexity in a controllable way, allowing for further trading off memory footprint and query latency. More generally, this line of research connects traditional IR models with neural “NLP” models and offers novel ways to explore the space between (efficient, but less effective) traditional rankers and (effective, but less efficient) neural rankers elegantly.
Förderung
- Netherlands Organization for Scientific Research
- Facebook Research
- Innovation Exchange Amsterdam
Zitieren
APA
Rau, D., Dehghani, M., & Kamps, J. (2024). Revisiting Bag of Words Document Representations for Efficient Ranking with Transformers. ACM Transactions on Information Systems, 42(5), 1-27. https://doi.org/10.1145/3640460
Vancouver
Rau D, Dehghani M, Kamps J. Revisiting Bag of Words Document Representations for Efficient Ranking with Transformers. ACM Transactions on Information Systems. 2024;42(5):1-27. doi:10.1145/3640460
BibTeX
@article{Rau2024,
author = {Rau, David and Dehghani, Mostafa and Kamps, Jaap},
title = {Revisiting Bag of Words Document Representations for Efficient Ranking with Transformers},
journal = {ACM Transactions on Information Systems},
year = {2024},
volume = {42},
number = {5},
pages = {1--27},
doi = {10.1145/3640460}
}
EUVIMED