Diffusion Recommender Models and the Illusion of Progress: A Concerning Study of Reproducibility and a Conceptual Mismatch
ACM Transactions on Recommender Systems · 30.09.2026 · 4 (3) · 1-69
DOI: 10.1145/3795792 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Recommender Systems, online erschienen am 16.03.2026. Verfasst von Michael Benigni und 2 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 81 Literaturangaben.
Abstract (Originalsprache)
Countless new machine learning models are published every year and are reported to significantly advance the state-of-the-art in top-n recommendation. However, earlier reproducibility studies indicate that progress in this area may be quite limited, due to widespread methodological issues, e.g., comparisons with untuned baseline models, creating an illusion of progress . In this work, we examine whether these problems persist in today’s research by attempting to reproduce nine SIGIR 2023 and 2024 recommendation algorithms based on Denoising Diffusion Probabilistic Models, a recent but rapidly expanding research area. Only 25% of reported results are fully reproducible and, since the original papers relied on weak baselines , they do not establish the superiority of diffusion models over state-of-the-art methods. In our controlled evaluations, well-tuned simpler baselines consistently exceed the diffusion-based models’ effectiveness reported in the original papers. Furthermore, we identify key mismatches between the characteristics of diffusion models and those of the traditional top-n recommendation task, raising doubts about their suitability for recommendation. Moreover, in the analyzed papers, the generative capabilities of these models are constrained to a minimum. Overall, our results call for greater scientific rigor and a disruptive change in the research and publication culture in this area.
Zitieren
APA
Benigni, M., Dacrema, M. F., & Jannach, D. (2026). Diffusion Recommender Models and the Illusion of Progress: A Concerning Study of Reproducibility and a Conceptual Mismatch. ACM Transactions on Recommender Systems, 4(3), 1-69. https://doi.org/10.1145/3795792
Vancouver
Benigni M, Dacrema MF, Jannach D. Diffusion Recommender Models and the Illusion of Progress: A Concerning Study of Reproducibility and a Conceptual Mismatch. ACM Transactions on Recommender Systems. 2026;4(3):1-69. doi:10.1145/3795792
BibTeX
@article{Benigni2026,
author = {Benigni, Michael and Dacrema, Maurizio Ferrari and Jannach, Dietmar},
title = {Diffusion Recommender Models and the Illusion of Progress: A Concerning Study of Reproducibility and a Conceptual Mismatch},
journal = {ACM Transactions on Recommender Systems},
year = {2026},
volume = {4},
number = {3},
pages = {1--69},
doi = {10.1145/3795792}
}
EUVIMED