EUVIMEDEUVIMED entdecken
Crossref

On Sampling Reliability in Offline Recommender Evaluation

Bruno Pereira, Alan Said, Rodrygo Santos

ACM Transactions on Recommender Systems · 13.11.2026

DOI: 10.1145/3862855 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Recommender Systems, online erschienen am 13.11.2026. Verfasst von Bruno Pereira und 2 weiteren Autor:innen. Die Arbeit nennt 61 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

When online testing is impractical or risky, offline evaluation remains the primary mechanism for benchmarking recommender systems. Yet it is inherently affected by two sources of distortion: exposure bias, arising from system-driven item visibility, and sampling bias, introduced when evaluation relies on a subset of logged items rather than the full catalog. Prior work has proposed techniques to mitigate sampling bias. However, these techniques are typically assessed on static logs. Such assessments do not establish whether the techniques support reliable model comparisons under varying exposure conditions or against true user preferences. In this paper, we systematically study how logging and sampling choices jointly affect the reliability of offline evaluation. Using a fully observed dataset as ground truth, we simulate diverse exposure regimes and evaluate common sampling strategies along four dimensions: resolution (model separability), fidelity (agreement with full evaluation), robustness (stability under exposure bias), and predictive power (alignment with ground truth). We further examine whether these criteria align or reveal trade-offs in sampler performance. To strengthen external validity, we replicate key analyses on two additional benchmark datasets with fixed missing-completely-at-random (MCAR) test exposure. Our findings characterize when sampling distorts evaluation outcomes and provide controlled empirical evidence regarding how sampling strategies behave under different exposure and sparsity conditions.

Zitieren

APA

Pereira, B., Said, A., & Santos, R. (2026). On Sampling Reliability in Offline Recommender Evaluation. ACM Transactions on Recommender Systems. https://doi.org/10.1145/3862855

Vancouver

Pereira B, Said A, Santos R. On Sampling Reliability in Offline Recommender Evaluation. ACM Transactions on Recommender Systems. 2026. doi:10.1145/3862855

BibTeX
@article{Pereira2026,
  author = {Pereira, Bruno and Said, Alan and Santos, Rodrygo},
  title = {On Sampling Reliability in Offline Recommender Evaluation},
  journal = {ACM Transactions on Recommender Systems},
  year = {2026},
  doi = {10.1145/3862855}
}
Kostenlos starten →