EUVIMEDEUVIMED entdecken
Crossref

A Critical Study on Data Leakage in Recommender System Offline Evaluation

Yitong Ji, Aixin Sun, Jie Zhang, Chenliang Li

ACM Transactions on Information Systems · 31.07.2023 · 41 (3) · 1-27

DOI: 10.1145/3569930 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Information Systems, online erschienen am 07.02.2023. Verfasst von Yitong Ji und 3 weiteren Autor:innen. Die Arbeit wurde bisher 90-mal zitiert (Crossref) und nennt 56 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recommender models are hard to evaluate, particularly under offline setting. In this article, we provide a comprehensive and critical analysis of the data leakage issue in recommender system offline evaluation. Data leakage is caused by not observing global timeline in evaluating recommenders e.g., train/test data split does not follow global timeline. As a result, a model learns from the user-item interactions that are not expected to be available at the prediction time. We first show the temporal dynamics of user-item interactions along global timeline, then explain why data leakage exists for collaborative filtering models. Through carefully designed experiments, we show that all models indeed recommend future items that are not available at the time point of a test instance, as the result of data leakage. The experiments are conducted with four widely used baseline models—BPR, NeuMF, SASRec, and LightGCN, on four popular offline datasets—MovieLens-25M, Yelp, Amazon-music, and Amazon-electronic, adopting leave-last-one-out data split. 1 We further show that data leakage does impact models’ recommendation accuracy. Their relative performance orders thus become unpredictable with different amount of leaked future data in training. To evaluate recommendation systems in a realistic manner in offline setting, we propose a timeline scheme, which calls for a revisit of the recommendation model design.

Zitieren

APA

Ji, Y., Sun, A., Zhang, J., & Li, C. (2023). A Critical Study on Data Leakage in Recommender System Offline Evaluation. ACM Transactions on Information Systems, 41(3), 1-27. https://doi.org/10.1145/3569930

Vancouver

Ji Y, Sun A, Zhang J, Li C. A Critical Study on Data Leakage in Recommender System Offline Evaluation. ACM Transactions on Information Systems. 2023;41(3):1-27. doi:10.1145/3569930

BibTeX
@article{Ji2023,
  author = {Ji, Yitong and Sun, Aixin and Zhang, Jie and Li, Chenliang},
  title = {A Critical Study on Data Leakage in Recommender System Offline Evaluation},
  journal = {ACM Transactions on Information Systems},
  year = {2023},
  volume = {41},
  number = {3},
  pages = {1--27},
  doi = {10.1145/3569930}
}
Kostenlos starten →