EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Now That Your System Has Been Reproduced, What Does This Mean for the Users?

Angelo Geninatti Cossatin, Timo Breuer, Noemi Mauro, Maria Maistro

ACM Transactions on Information Systems · 30.06.2026 · 44 (5) · 1-41

DOI: 10.1145/3814609 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Information Systems, online erschienen am 11.06.2026. Verfasst von Angelo Geninatti Cossatin und 3 weiteren Autor:innen. Die Arbeit nennt 137 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: FH-Personal funding programme, project PLan_CV, German Federal Ministry of Education and Research, Joint Science Conference.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Reproducibility lies at the basis of the empirical method: a novel approach will be widely adopted if its experimental results can be validated and reproduced by the community. Previous work on reproducibility in Information Retrieval (IR) has mainly addressed the reproducibility and replicability of offline experiments, with a few exceptions that replicate user studies. To the best of our knowledge, no previous work has investigated how reproducibility affects real users. In this paper, we do that by evaluating and comparing the reproducibility of an IR system both offline and online. We consider a reference system and generate a constellation of reproduced systems with varying parameters. We select 6 systems with different degrees of offline reproducibility. We then run a between-subjects online experiment with 280 participants and collect clicks to evaluate online reproducibility. Results show that real users do not perceive moderate variations of the reproducibility degree of systems, while they become relevant when the difference with the original system increases. Furthermore, we trained a click model to evaluate online reproducibility with simulated clicks. Results are not consistent with those from the user study, suggesting that better click models are needed to evaluate online reproducibility. Our data and source code are publicly available: https://github.com/angelogeninatti/reproducibilityLogs .

Förderung

  • FH-Personal funding programme, project PLan_CV
  • German Federal Ministry of Education and Research
  • Joint Science Conference

Zitieren

APA

Cossatin, A. G., Breuer, T., Mauro, N., & Maistro, M. (2026). Now That Your System Has Been Reproduced, What Does This Mean for the Users?. ACM Transactions on Information Systems, 44(5), 1-41. https://doi.org/10.1145/3814609

Vancouver

Cossatin AG, Breuer T, Mauro N, Maistro M. Now That Your System Has Been Reproduced, What Does This Mean for the Users?. ACM Transactions on Information Systems. 2026;44(5):1-41. doi:10.1145/3814609

BibTeX
@article{Cossatin2026,
  author = {Cossatin, Angelo Geninatti and Breuer, Timo and Mauro, Noemi and Maistro, Maria},
  title = {Now That Your System Has Been Reproduced, What Does This Mean for the Users?},
  journal = {ACM Transactions on Information Systems},
  year = {2026},
  volume = {44},
  number = {5},
  pages = {1--41},
  doi = {10.1145/3814609}
}
Kostenlos starten →