Understanding and Predicting User Satisfaction with Conversational Recommender Systems
ACM Transactions on Information Systems · 31.03.2024 · 42 (2) · 1-37
DOI: 10.1145/3624989 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Information Systems, online erschienen am 08.11.2023. Verfasst von Clemencia Siro und 2 weiteren Autor:innen. Die Arbeit wurde bisher 24-mal zitiert (Crossref) und nennt 75 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-ND 4.0).
Abstract (Originalsprache)
User satisfaction depicts the effectiveness of a system from the user’s perspective. Understanding and predicting user satisfaction is vital for the design of user-oriented evaluation methods for conversational recommender systems (CRSs) . Current approaches rely on turn-level satisfaction ratings to predict a user’s overall satisfaction with CRS. These methods assume that all users perceive satisfaction similarly, failing to capture the broader dialogue aspects that influence overall user satisfaction. We investigate the effect of several dialogue aspects on user satisfaction when interacting with a CRS. To this end, we annotate dialogues based on six aspects (i.e., relevance , interestingness , understanding , task-completion , interest-arousal , and efficiency ) at the turn and dialogue levels. We find that the concept of satisfaction varies per user. At the turn level, a system’s ability to make relevant recommendations is a significant factor in satisfaction. We adopt these aspects as features for predicting response quality and user satisfaction. We achieve an F1-score of 0.80 in classifying dissatisfactory dialogues, and a Pearson’s r of 0.73 for turn-level response quality estimation, demonstrating the effectiveness of the proposed dialogue aspects in predicting user satisfaction and being able to identify dialogues where the system is failing. With this article, we release our annotated data. 1
Zitieren
APA
Siro, C., Aliannejadi, M., & Rijke, M. D. (2024). Understanding and Predicting User Satisfaction with Conversational Recommender Systems. ACM Transactions on Information Systems, 42(2), 1-37. https://doi.org/10.1145/3624989
Vancouver
Siro C, Aliannejadi M, Rijke MD. Understanding and Predicting User Satisfaction with Conversational Recommender Systems. ACM Transactions on Information Systems. 2024;42(2):1-37. doi:10.1145/3624989
BibTeX
@article{Siro2024,
author = {Siro, Clemencia and Aliannejadi, Mohammad and Rijke, Maarten De},
title = {Understanding and Predicting User Satisfaction with Conversational Recommender Systems},
journal = {ACM Transactions on Information Systems},
year = {2024},
volume = {42},
number = {2},
pages = {1--37},
doi = {10.1145/3624989}
}
EUVIMED