EUVIMEDEUVIMED entdecken
Crossref

Anytime-valid off-policy Inference for Contextual Bandits

Ian Waudby-Smith, Lili Wu, Aaditya Ramdas, Nikos Karampatziakis, Paul Mineiro

ACM / IMS Journal of Data Science · 2024

DOI: 10.1145/3643693 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM / IMS Journal of Data Science, 2024. Verfasst von Ian Waudby-Smith und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 65 Literaturangaben. Gefördert von: NSF, ARL IoBT REIGN, DEVCOM Army Research Laboratory under Cooperative Agreement.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Contextual bandit algorithms are ubiquitous tools for active sequential experimentation in healthcare and the tech industry. They involve online learning algorithms that adaptively learn policies over time to map observed contexts X t to actions A t in an attempt to maximize stochastic rewards R t . This adaptivity raises interesting but hard statistical inference questions, especially counterfactual ones: for example, it is often of interest to estimate the properties of a hypothetical policy that is different from the logging policy that was used to collect the data—a problem known as “off-policy evaluation” (OPE). Using modern martingale techniques, we present a comprehensive framework for OPE inference that relaxes unnecessary conditions made in some past works (such as performing inference at prespecified sample sizes, uniformly bounded importance weights, constant logging policies, and constant policy values, among others), significantly improving on them both theoretically and empirically. Importantly, our methods can be employed while the original experiment is still running (that is, not necessarily post hoc), when the logging policy may be itself changing (due to learning), and even if the context distributions are a highly dependent time series (such as if they are drifting over time). More concretely, we derive confidence sequences for various functionals of interest in OPE. These include doubly robust ones for time-varying off-policy mean reward values, but also confidence bands for the entire cumulative distribution function of the off-policy reward distribution. All of our methods (a) are valid at arbitrary stopping times; (b) only make nonparametric assumptions; (c) do not require importance weights to be uniformly bounded, and if they are, we do not need to know these bounds; and (d) adapt to the empirical variance of our estimators. In summary, our methods enable anytime-valid off-policy inference using adaptively collected contextual bandit data.

Förderung

  • NSF
  • ARL IoBT REIGN
  • DEVCOM Army Research Laboratory under Cooperative Agreement

Zitieren

APA

Waudby-Smith, I., Wu, L., Ramdas, A., Karampatziakis, N., & Mineiro, P. (2024). Anytime-valid off-policy Inference for Contextual Bandits. ACM / IMS Journal of Data Science. https://doi.org/10.1145/3643693

Vancouver

Waudby-Smith I, Wu L, Ramdas A, Karampatziakis N, Mineiro P. Anytime-valid off-policy Inference for Contextual Bandits. ACM / IMS Journal of Data Science. 2024. doi:10.1145/3643693

BibTeX
@article{WaudbySmith2024,
  author = {Waudby-Smith, Ian and Wu, Lili and Ramdas, Aaditya and Karampatziakis, Nikos and Mineiro, Paul},
  title = {Anytime-valid off-policy Inference for Contextual Bandits},
  journal = {ACM / IMS Journal of Data Science},
  year = {2024},
  doi = {10.1145/3643693}
}
Kostenlos starten →