Top Challenges from the first Practical Online Controlled Experiments Summit
ACM SIGKDD Explorations Newsletter · 13.05.2019 · 21 (1) · 20-35
DOI: 10.1145/3331651.3331655 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM SIGKDD Explorations Newsletter, online erschienen am 13.05.2019. Verfasst von Somit Gupta und 33 weiteren Autor:innen. Die Arbeit wurde bisher 95-mal zitiert (Crossref) und nennt 77 Literaturangaben.
Abstract (Originalsprache)
Online controlled experiments (OCEs), also known as A/B tests, have become ubiquitous in evaluating the impact of changes made to software products and services. While the concept of online controlled experiments is simple, there are many practical challenges in running OCEs at scale. To understand the top practical challenges in running OCEs at scale and encourage further academic and industrial exploration, representatives with experience in large-scale experimentation from thirteen different organizations (Airbnb, Amazon, Booking.com, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, Yandex, and Stanford University) were invited to the first Practical Online Controlled Experiments Summit. All thirteen organizations sent representatives. Together these organizations have tested more than one hundred thousand experiment treatments last year. Thirty-four experts from these organizations participated in the summit in Sunnyvale, CA, USA on December 13-14, 2018. While there are papers from individual organizations on some of the challenges and pitfalls in running OCEs at scale, this is the first paper to provide the top challenges faced across the industry for running OCEs at scale and some common solutions.
Zitieren
APA
Gupta, S., Kohavi, R., Tang, D., Xu, Y., Andersen, R., Bakshy, E., Cardin, N., Chandran, S., Chen, N., Coey, D., Curtis, M., Deng, A., Duan, W., Forbes, P., Frasca, B., Guy, T., Imbens, G. W., Jacques, G. S., Kantawala, P., …, & Yashkov, I. (2019). Top Challenges from the first Practical Online Controlled Experiments Summit. ACM SIGKDD Explorations Newsletter, 21(1), 20-35. https://doi.org/10.1145/3331651.3331655
Vancouver
Gupta S, Kohavi R, Tang D, Xu Y, Andersen R, Bakshy E, et al. Top Challenges from the first Practical Online Controlled Experiments Summit. ACM SIGKDD Explorations Newsletter. 2019;21(1):20-35. doi:10.1145/3331651.3331655
BibTeX
@article{Gupta2019,
author = {Gupta, Somit and Kohavi, Ronny and Tang, Diane and Xu, Ya and Andersen, Reid and Bakshy, Eytan and Cardin, Niall and Chandran, Sumita and Chen, Nanyu and Coey, Dominic and Curtis, Mike and Deng, Alex and Duan, Weitao and Forbes, Peter and Frasca, Brian and Guy, Tommy and Imbens, Guido W. and Jacques, Guillaume Saint and Kantawala, Pranav and Katsev, Ilya and Katzwer, Moshe and Konutgan, Mikael and Kunakova, Elena and Lee, Minyong and Lee, MJ and Liu, Joseph and McQueen, James and Najmi, Amir and Smith, Brent and Trehan, Vivek and Vermeer, Lukas and Walker, Toby and Wong, Jeffrey and Yashkov, Igor},
title = {Top Challenges from the first Practical Online Controlled Experiments Summit},
journal = {ACM SIGKDD Explorations Newsletter},
year = {2019},
volume = {21},
number = {1},
pages = {20--35},
doi = {10.1145/3331651.3331655}
}
EUVIMED