EUVIMEDEUVIMED entdecken
Crossref

Online and Offline Evaluation in Search Clarification

Leila Tavakoli, Johanne R. Trippas, Hamed Zamani, Falk Scholer, Mark Sanderson

ACM Transactions on Information Systems · 31.01.2025 · 43 (1) · 1-30

DOI: 10.1145/3681786 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Information Systems, online erschienen am 04.11.2024. Verfasst von Leila Tavakoli und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 81 Literaturangaben. Gefördert von: Australian Research Council, Office of Naval Research contract, NSF grant number.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The effectiveness of clarification question models in engaging users within search systems is currently constrained, casting doubt on their overall usefulness. To improve the performance of these models, it is crucial to employ assessment approaches that encompass both real-time feedback from users (online evaluation) and the characteristics of clarification questions evaluated through human assessment (offline evaluation). However, the relationship between online and offline evaluations has been debated in information retrieval. This study aims to investigate how this discordance holds in search clarification. We use user engagement as ground truth and employ several offline labels to investigate to what extent the offline ranked lists of clarification resemble the ideal ranked lists based on online user engagement. Contrary to the current understanding that offline evaluations fall short of supporting online evaluations, we indicate that when identifying the most engaging clarification questions from the user’s perspective, online and offline evaluations correspond with each other. We show that the query length does not influence the relationship between online and offline evaluations, and reducing uncertainty in online evaluation strengthens this relationship. We illustrate that an engaging clarification needs to excel from multiple perspectives, and SERP quality and characteristics of the clarification are equally important. We also investigate if human labels can enhance the performance of Large Language Models (LLMs) and Learning-to-Rank (LTR) models in identifying the most engaging clarification questions from the user’s perspective by incorporating offline evaluations as input features. Our results indicate that LTR models do not perform better than individual offline labels. However, GPT, an LLM, emerges as the standout performer, surpassing all LTR models and offline labels.

Förderung

  • Australian Research Council
  • Office of Naval Research contract
  • NSF grant number

Zitieren

APA

Tavakoli, L., Trippas, J. R., Zamani, H., Scholer, F., & Sanderson, M. (2025). Online and Offline Evaluation in Search Clarification. ACM Transactions on Information Systems, 43(1), 1-30. https://doi.org/10.1145/3681786

Vancouver

Tavakoli L, Trippas JR, Zamani H, Scholer F, Sanderson M. Online and Offline Evaluation in Search Clarification. ACM Transactions on Information Systems. 2025;43(1):1-30. doi:10.1145/3681786

BibTeX
@article{Tavakoli2025,
  author = {Tavakoli, Leila and Trippas, Johanne R. and Zamani, Hamed and Scholer, Falk and Sanderson, Mark},
  title = {Online and Offline Evaluation in Search Clarification},
  journal = {ACM Transactions on Information Systems},
  year = {2025},
  volume = {43},
  number = {1},
  pages = {1--30},
  doi = {10.1145/3681786}
}
Kostenlos starten →