An Empirical Study on Challenges for LLM Application Developers
ACM Transactions on Software Engineering and Methodology · 30.09.2025 · 34 (7) · 1-37
DOI: 10.1145/3715007 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 17.08.2025. Verfasst von Xiang Chen und 4 weiteren Autor:innen. Die Arbeit wurde bisher 40-mal zitiert (Crossref) und nennt 74 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Postgraduate Research & Practice Innovation Program of Jiangsu Province.
Abstract (Originalsprache)
In recent years, large language models (LLMs) have seen rapid advancements, significantly impacting various fields such as computer vision, natural language processing, and software engineering. These LLMs, exemplified by OpenAI's ChatGPT, have revolutionized the way we approach language understanding and generation tasks. However, in contrast to traditional software development practices, LLM development introduces new challenges for AI developers in design, implementation, and deployment. These challenges span different areas (such as prompts, APIs, and plugins), requiring developers to navigate unique methodologies and considerations specific to LLM application development. Despite the profound influence of LLMs, to the best of our knowledge, these challenges have not been thoroughly investigated in previous empirical studies. To fill this gap, we present the first comprehensive study on understanding the challenges faced by LLM developers. Specifically, we crawl and analyze 29,057 relevant questions from a popular OpenAI developer forum. We first examine their popularity and difficulty. After manually analyzing 2,364 sampled questions, we construct a taxonomy of challenges faced by LLM developers. Based on this taxonomy, we summarize a set of findings and actionable implications for LLM-related stakeholders, including developers and providers (especially the OpenAI organization).
Förderung
- National Natural Science Foundation of China
- Postgraduate Research & Practice Innovation Program of Jiangsu Province
Zitieren
APA
Chen, X., Gao, C., Chen, C., Zhang, G., & Liu, Y. (2025). An Empirical Study on Challenges for LLM Application Developers. ACM Transactions on Software Engineering and Methodology, 34(7), 1-37. https://doi.org/10.1145/3715007
Vancouver
Chen X, Gao C, Chen C, Zhang G, Liu Y. An Empirical Study on Challenges for LLM Application Developers. ACM Transactions on Software Engineering and Methodology. 2025;34(7):1-37. doi:10.1145/3715007
BibTeX
@article{Chen2025,
author = {Chen, Xiang and Gao, Chaoyang and Chen, Chunyang and Zhang, Guangbei and Liu, Yong},
title = {An Empirical Study on Challenges for LLM Application Developers},
journal = {ACM Transactions on Software Engineering and Methodology},
year = {2025},
volume = {34},
number = {7},
pages = {1--37},
doi = {10.1145/3715007}
}
EUVIMED