A Revisiting Study of Appropriate Offline Evaluation for Top- N Recommendation Algorithms
ACM Transactions on Information Systems · 2023-04-30 · 41 (2) · 1-41
DOI: 10.1145/3545796 ↗At a glance
Journal article in ACM Transactions on Information Systems, published online on 2022-12-21. Written by Wayne Xin Zhao and 4 co-authors. Cited 31 times so far (Crossref); 140 references. Funded by: National Natural Science Foundation of China, Beijing Natural Science Foundation, Beijing Outstanding Young Scientist Program.
Abstract (original language)
In recommender systems, top- N recommendation is an important task with implicit feedback data. Although the recent success of deep learning largely pushes forward the research on top- N recommendation, there are increasing concerns on appropriate evaluation of recommendation algorithms. It therefore is important to study how recommendation algorithms can be reliably evaluated and thoroughly verified. This work presents a large-scale, systematic study on six important factors from three aspects for evaluating recommender systems. We carefully select 12 top- N recommendation algorithms and eight recommendation datasets. Our experiments are carefully designed and extensively conducted with these algorithms and datasets. In particular, all the experiments in our work are implemented based on an open sourced recommendation library, Recbole [ 139 ], which ensures the reproducibility and reliability of our results. Based on the large-scale experiments and detailed analysis, we derive several key findings on the experimental settings for evaluating recommender systems. Our findings show that some settings can lead to substantial or significant differences in performance ranking of the compared algorithms. In response to recent evaluation concerns, we also provide several suggested settings that are specially important for performance comparison.
Funding
- National Natural Science Foundation of China
- Beijing Natural Science Foundation
- Beijing Outstanding Young Scientist Program
- Beijing Academy of Artificial Intelligence
Cite
APA
Zhao, W. X., Lin, Z., Feng, Z., Wang, P., & Wen, J. (2023). A Revisiting Study of Appropriate Offline Evaluation for Top- N Recommendation Algorithms. ACM Transactions on Information Systems, 41(2), 1-41. https://doi.org/10.1145/3545796
Vancouver
Zhao WX, Lin Z, Feng Z, Wang P, Wen J. A Revisiting Study of Appropriate Offline Evaluation for Top- N Recommendation Algorithms. ACM Transactions on Information Systems. 2023;41(2):1-41. doi:10.1145/3545796
BibTeX
@article{Zhao2023,
author = {Zhao, Wayne Xin and Lin, Zihan and Feng, Zhichao and Wang, Pengfei and Wen, Ji-Rong},
title = {A Revisiting Study of Appropriate Offline Evaluation for Top- N Recommendation Algorithms},
journal = {ACM Transactions on Information Systems},
year = {2023},
volume = {41},
number = {2},
pages = {1--41},
doi = {10.1145/3545796}
}
EUVIMED