On the Effectiveness of Large Language Models in Domain-Specific Code Generation
ACM Transactions on Software Engineering and Methodology · 31.03.2025 · 34 (3) · 1-22
DOI: 10.1145/3697012 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 23.02.2025. Verfasst von Xiaodong Gu und 8 weiteren Autor:innen. Die Arbeit wurde bisher 55-mal zitiert (Crossref) und nennt 56 Literaturangaben. Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China, CCF-Tencent Open Research Fund.
Abstract (Originalsprache)
Large language models (LLMs) such as ChatGPT have shown remarkable capabilities in code generation. Despite significant achievements, they rely on enormous training data to acquire a broad spectrum of open-domain knowledge. Besides, their evaluation revolves around open-domain benchmarks like HumanEval, which primarily consist of programming contests. Therefore, it is hard to fully characterize the intricacies and challenges associated with particular domains (e.g., Web, game, and math). In this article, we conduct an in-depth study of the LLMs in domain-specific code generation. Our results demonstrate that LLMs exhibit sub-optimal performance in generating domain-specific code, due to their limited proficiency in utilizing domain-specific libraries. We further observe that incorporating API knowledge as prompts can empower LLMs to generate more professional code. Based on these findings, we further investigate how to effectively incorporate API knowledge into the code generation process. We experiment with three strategies for incorporating domain knowledge, namely, external knowledge inquirer, chain-of-thought prompting, and chain-of-thought fine-tuning. We refer to these strategies as a new code generation approach called DomCoder . Experimental results show that all strategies of DomCoder improve the effectiveness of domain-specific code generation under certain settings.
Förderung
- National Key R&D Program of China
- National Natural Science Foundation of China
- CCF-Tencent Open Research Fund
Zitieren
APA
Gu, X., Chen, M., Lin, Y., Hu, Y., Zhang, H., Wan, C., Wei, Z., Xu, Y., & Wang, J. (2025). On the Effectiveness of Large Language Models in Domain-Specific Code Generation. ACM Transactions on Software Engineering and Methodology, 34(3), 1-22. https://doi.org/10.1145/3697012
Vancouver
Gu X, Chen M, Lin Y, Hu Y, Zhang H, Wan C, et al. On the Effectiveness of Large Language Models in Domain-Specific Code Generation. ACM Transactions on Software Engineering and Methodology. 2025;34(3):1-22. doi:10.1145/3697012
BibTeX
@article{Gu2025,
author = {Gu, Xiaodong and Chen, Meng and Lin, Yalan and Hu, Yuhan and Zhang, Hongyu and Wan, Chengcheng and Wei, Zhao and Xu, Yong and Wang, Juhong},
title = {On the Effectiveness of Large Language Models in Domain-Specific Code Generation},
journal = {ACM Transactions on Software Engineering and Methodology},
year = {2025},
volume = {34},
number = {3},
pages = {1--22},
doi = {10.1145/3697012}
}
EUVIMED