AI-Assisted Code Authoring at Scale: Fine-Tuning, Deploying, and Mixed Methods Evaluation
Proceedings of the ACM on Software Engineering · 12.07.2024 · 1 (FSE) · 1066-1085
DOI: 10.1145/3643774 ↗Kurz gefasst
Mixed-Methods-Studie in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 12.07.2024. Verfasst von Vijayaraghavan Murali und 8 weiteren Autor:innen. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 39 Literaturangaben.
Abstract (Originalsprache)
Generative LLMs have been shown to effectively power AI-based code authoring tools that can suggest entire statements or blocks of code during code authoring. In this paper we present CodeCompose , an AI-assisted code authoring tool developed and deployed at Meta internally. CodeCompose is based on the InCoder LLM that merges generative capabilities with bi-directionality. We have scaled up CodeCompose to serve tens of thousands of developers at Meta, across 9 programming languages and several coding surfaces. We present our experience in making design decisions about the model and system architecture for CodeCompose that addresses these challenges. To release a LLM model at this scale, we needed to first ensure that it is sufficiently accurate. In a random sample of 20K source code files, depending on the language, we are able to reproduce hidden lines between 40% and 58% of the time, an improvement of 1.4× and 4.1× over a model trained only on public data. We gradually rolled CodeCompose out to developers. At the time of this writing, 16K developers have used it with 8% of their code coming directly from CodeCompose . To triangulate our numerical findings, we conduct a thematic analysis on the feedback from 70 developers. We find that 91.5% of the feedback is positive, with the most common themes being discovering APIs, dealing with boilerplate code, and accelerating coding. Meta continues to integrate this feedback into CodeCompose .
Zitieren
APA
Murali, V., Maddila, C., Ahmad, I., Bolin, M., Cheng, D., Ghorbani, N., Fernandez, R., Nagappan, N., & Rigby, P. C. (2024). AI-Assisted Code Authoring at Scale: Fine-Tuning, Deploying, and Mixed Methods Evaluation. Proceedings of the ACM on Software Engineering, 1(FSE), 1066-1085. https://doi.org/10.1145/3643774
Vancouver
Murali V, Maddila C, Ahmad I, Bolin M, Cheng D, Ghorbani N, et al. AI-Assisted Code Authoring at Scale: Fine-Tuning, Deploying, and Mixed Methods Evaluation. Proceedings of the ACM on Software Engineering. 2024;1(FSE):1066-1085. doi:10.1145/3643774
BibTeX
@article{Murali2024,
author = {Murali, Vijayaraghavan and Maddila, Chandra and Ahmad, Imad and Bolin, Michael and Cheng, Daniel and Ghorbani, Negar and Fernandez, Renuka and Nagappan, Nachiappan and Rigby, Peter C.},
title = {AI-Assisted Code Authoring at Scale: Fine-Tuning, Deploying, and Mixed Methods Evaluation},
journal = {Proceedings of the ACM on Software Engineering},
year = {2024},
volume = {1},
number = {FSE},
pages = {1066--1085},
doi = {10.1145/3643774}
}
EUVIMED