EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning

Shreya Shankar, Rolando Garcia, Joseph M. Hellerstein, Aditya G. Parameswaran

Proceedings of the ACM on Human-Computer Interaction · 17.04.2024 · 8 (CSCW1) · 1-34

DOI: 10.1145/3653697 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Human-Computer Interaction, online erschienen am 26.04.2024. Verfasst von Shreya Shankar und 3 weiteren Autor:innen. Die Arbeit wurde bisher 30-mal zitiert (Crossref) und nennt 111 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Organizations rely on machine learning engineers (MLEs) to deploy models and maintain ML pipelines in production. Due to models' extensive reliance on fresh data, the operationalization of machine learning, or MLOps, requires MLEs to have proficiency in data science and engineering. When considered holistically, the job seems staggering---how do MLEs do MLOps, and what are their unaddressed challenges? To address these questions, we conducted semi-structured ethnographic interviews with 18 MLEs working on various applications, including chatbots, autonomous vehicles, and finance. We find that MLEs engage in a workflow of (i) data preparation, (ii) experimentation, (iii) evaluation throughout a multi-staged deployment, and (iv) continual monitoring and response. Throughout this workflow, MLEs collaborate extensively with data scientists, product stakeholders, and one another, supplementing routine verbal exchanges with communication tools ranging from Slack to organization-wide ticketing and reporting systems. We introduce the 3Vs of MLOps: velocity, visibility, and versioning --- three virtues of successful ML deployments that MLEs learn to balance and grow as they mature. Finally, we discuss design implications and opportunities for future work.

Förderung

  • National Science Foundation

Zitieren

APA

Shankar, S., Garcia, R., Hellerstein, J. M., & Parameswaran, A. G. (2024). "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW1), 1-34. https://doi.org/10.1145/3653697

Vancouver

Shankar S, Garcia R, Hellerstein JM, Parameswaran AG. "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning. Proceedings of the ACM on Human-Computer Interaction. 2024;8(CSCW1):1-34. doi:10.1145/3653697

BibTeX
@article{Shankar2024,
  author = {Shankar, Shreya and Garcia, Rolando and Hellerstein, Joseph M. and Parameswaran, Aditya G.},
  title = {"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning},
  journal = {Proceedings of the ACM on Human-Computer Interaction},
  year = {2024},
  volume = {8},
  number = {CSCW1},
  pages = {1--34},
  doi = {10.1145/3653697}
}
Kostenlos starten →