GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents
ACM Transactions on Software Engineering and Methodology · 30.11.2026 · 35 (11) · 1-32
DOI: 10.1145/3815588 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 03.11.2026. Verfasst von Jie JW Wu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Natural Sciences and Engineering Research Council of Canada.
Abstract (Originalsprache)
With data-driven development now widely adopted in the software industry, online A/B testing is an established method for measuring the effects of new technologies. However, deploying online experiments demands considerable resources for experimental design, implementation, and production deployment and may negatively impact users (e.g., unsafe or unethical outcomes) while requiring multiple weeks of data collection. To tackle these issues, a growing research area known as off-policy evaluation (OPE) (offline A/B testing) has gained significant attention. Its goal is to assess new technologies offline by leveraging previously collected logged data. OPE is also a fundamental problem in reinforcement learning (RL) and is important in areas where online A/B testing is expensive or risky, such as healthcare, recommender systems, education, dialog systems, and robotics. Despite recent advances in code-generation large language models (LLMs) and agentic workflows, little is known about whether and how LLMs and LLM-based agents can be leveraged to automatically optimize OPE implementations. In this work, we propose GrowthHacker , a benchmark that evaluates baseline LLMs and LLM-based agents on large-scale public datasets. GrowthHacker autonomously and iteratively modifies code, runs OPE, and uses the resulting metrics to guide subsequent rounds of optimization. We evaluate methods on Open Bandit Pipeline and Scope-RL, and we develop a two_agent framework that addresses limitations of existing agent frameworks while reducing system complexity. Across both libraries, two_agent demonstrates the highest reliability (98.1–100% success rate) and the highest positive-outcome rate (78%), with a median improvement of 4.4% among positive outcomes; CrewAI achieves the highest average improvement (37.9%) and is the only framework with zero extreme-value failures. AutoGen and Default each reach 65% positive-outcome rates. These results establish the feasibility of using LLM-based agents as automated “growth hackers” to continuously improve OPE systems, with implications for scaling data-driven decision-making in production environments where manual optimization is expensive. Our replication package is available at https://doi.org/10.5281/zenodo.17496869 and https://github.com/jie-jw-wu/ope-agent .
Förderung
- Natural Sciences and Engineering Research Council of Canada
Zitieren
APA
Wu, J. J., Herlihy, A. P., Mirza, A. S., Afoud, A., & Fard, F. (2026). GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents. ACM Transactions on Software Engineering and Methodology, 35(11), 1-32. https://doi.org/10.1145/3815588
Vancouver
Wu JJ, Herlihy AP, Mirza AS, Afoud A, Fard F. GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents. ACM Transactions on Software Engineering and Methodology. 2026;35(11):1-32. doi:10.1145/3815588
BibTeX
@article{Wu2026,
author = {Wu, Jie JW and Herlihy, Ayanda Patrick and Mirza, Ahmad Saleem and Afoud, Ali and Fard, Fatemeh},
title = {GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents},
journal = {ACM Transactions on Software Engineering and Methodology},
year = {2026},
volume = {35},
number = {11},
pages = {1--32},
doi = {10.1145/3815588}
}
EUVIMED