Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing
ACM Transactions on Information Systems · 31.01.2026 · 44 (1) · 1-43
DOI: 10.1145/3772275 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Information Systems, online erschienen am 15.12.2025. Verfasst von Maurizio Ferrari Dacrema und 2 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 83 Literaturangaben.
Abstract (Originalsprache)
Graph-based techniques relying on neural networks and embeddings have gained attention as a way to develop Recommender Systems (RS) with several papers on the topic presented at SIGIR 2022 and 2023. Given the importance of ensuring that published research is methodologically sound and reproducible, in this paper, we analyze 10 graph-based RS papers, most of which were published at SIGIR 2022, and assess their impact on subsequent work published in SIGIR 2023. Our analysis reveals several critical points that require attention: (i) the prevalence of bad practices, such as erroneous data splits or information leakage between training and testing data, which call into question the validity of the results; (ii) frequent inconsistencies between the provided artifacts (source code and data) and their descriptions in the paper, causing uncertainty about what is actually being evaluated; and (iii) the preference for new or complex baselines that are weaker compared to simpler ones, creating the impression of continuous improvement even when, particularly for the Amazon-Book dataset, the state-of-the-art has significantly worsened. Due to these issues, we are unable to confirm the claims made in most of the papers that we examined and attempted to reproduce.
Zitieren
APA
Dacrema, M. F., Benigni, M., & Ferro, N. (2026). Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing. ACM Transactions on Information Systems, 44(1), 1-43. https://doi.org/10.1145/3772275
Vancouver
Dacrema MF, Benigni M, Ferro N. Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing. ACM Transactions on Information Systems. 2026;44(1):1-43. doi:10.1145/3772275
BibTeX
@article{Dacrema2026,
author = {Dacrema, Maurizio Ferrari and Benigni, Michael and Ferro, Nicola},
title = {Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing},
journal = {ACM Transactions on Information Systems},
year = {2026},
volume = {44},
number = {1},
pages = {1--43},
doi = {10.1145/3772275}
}
EUVIMED