EUVIMEDEUVIMED entdecken
Crossref

DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task

Wenhan Liu, Yutao Zhu, Zhicheng Dou, Yujia Zhou

ACM Transactions on Information Systems · 31.03.2026 · 44 (3) · 1-25

DOI: 10.1145/3788867 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Information Systems, online erschienen am 02.03.2026. Verfasst von Wenhan Liu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 49 Literaturangaben. Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) have been proven to have strong zero-shot passage ranking capabilities. In-context learning effectively enhances LLM performance by providing few-shot demonstrations, opening avenues for further improving LLM’s ranking ability. However, existing studies usually retrieve the most similar demonstrations to the input, ignoring the demonstration dependencies and diversity, which is insufficient to inspire the LLM for assessing the current query-passage relevance. In this article, we propose a framework named DemoRank, which selects few-shot demonstrations by performing a novel dependency-aware reranking of the retrieved demonstrations. Considering the dependency, combining top-ranked demonstrations yields better results. Nevertheless, generating the training samples for such a dependency-aware demonstration reranker faces two challenges: (1) the traditional demonstration ranked list assumes demonstration independence, which cannot be used to train our reranker, and (2) obtaining the optimal demonstration ranked list from the retrieved set is NP-hard and inefficient. To overcome these challenges, we propose an approach to construct a kind of dependency-aware training samples efficiently and design a list-pairwise training approach for the optimization of the demonstration reranker. We conduct extensive experiments on a series of passage ranking datasets, and the results demonstrate the superior performance of our proposed DemoRank framework under various scenarios. Our code is publicly available at https://github.com/8421bcd/demorank .

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Liu, W., Zhu, Y., Dou, Z., & Zhou, Y. (2026). DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task. ACM Transactions on Information Systems, 44(3), 1-25. https://doi.org/10.1145/3788867

Vancouver

Liu W, Zhu Y, Dou Z, Zhou Y. DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task. ACM Transactions on Information Systems. 2026;44(3):1-25. doi:10.1145/3788867

BibTeX
@article{Liu2026,
  author = {Liu, Wenhan and Zhu, Yutao and Dou, Zhicheng and Zhou, Yujia},
  title = {DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task},
  journal = {ACM Transactions on Information Systems},
  year = {2026},
  volume = {44},
  number = {3},
  pages = {1--25},
  doi = {10.1145/3788867}
}
Kostenlos starten →