EUVIMEDEUVIMED entdecken
Crossref

Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing

Maurizio Ferrari Dacrema, Michael Benigni, Nicola Ferro

ACM Transactions on Information Systems · 31.01.2026 · 44 (1) · 1-43

DOI: 10.1145/3772275 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Information Systems, online erschienen am 15.12.2025. Verfasst von Maurizio Ferrari Dacrema und 2 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 83 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Graph-based techniques relying on neural networks and embeddings have gained attention as a way to develop Recommender Systems (RS) with several papers on the topic presented at SIGIR 2022 and 2023. Given the importance of ensuring that published research is methodologically sound and reproducible, in this paper, we analyze 10 graph-based RS papers, most of which were published at SIGIR 2022, and assess their impact on subsequent work published in SIGIR 2023. Our analysis reveals several critical points that require attention: (i) the prevalence of bad practices, such as erroneous data splits or information leakage between training and testing data, which call into question the validity of the results; (ii) frequent inconsistencies between the provided artifacts (source code and data) and their descriptions in the paper, causing uncertainty about what is actually being evaluated; and (iii) the preference for new or complex baselines that are weaker compared to simpler ones, creating the impression of continuous improvement even when, particularly for the Amazon-Book dataset, the state-of-the-art has significantly worsened. Due to these issues, we are unable to confirm the claims made in most of the papers that we examined and attempted to reproduce.

Zitieren

APA

Dacrema, M. F., Benigni, M., & Ferro, N. (2026). Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing. ACM Transactions on Information Systems, 44(1), 1-43. https://doi.org/10.1145/3772275

Vancouver

Dacrema MF, Benigni M, Ferro N. Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing. ACM Transactions on Information Systems. 2026;44(1):1-43. doi:10.1145/3772275

BibTeX
@article{Dacrema2026,
  author = {Dacrema, Maurizio Ferrari and Benigni, Michael and Ferro, Nicola},
  title = {Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing},
  journal = {ACM Transactions on Information Systems},
  year = {2026},
  volume = {44},
  number = {1},
  pages = {1--43},
  doi = {10.1145/3772275}
}
Kostenlos starten →