EUVIMEDOpdag EUVIMED
Open access · CC BY 4.0

An Empirical Study of the Non-Determinism of ChatGPT in Code Generation

Shuyin Ouyang, Jie M. Zhang, Mark Harman, Meng Wang

ACM Transactions on Software Engineering and Methodology · 2025-02-28 · 34 (2) · 1-28

DOI: 10.1145/3697010 ↗

At a glance

Journal article in ACM Transactions on Software Engineering and Methodology, published online on 2025-01-22. Written by Shuyin Ouyang and 3 co-authors. Cited 245 times so far (Crossref); 73 references. The full text is freely available (open access, CC BY 4.0). Funded by: UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence.

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; non-deterministically returning very different code for the same prompt. Such non-determinism affects the correctness and consistency of the generated code, undermines developers’ trust in LLMs, and yields low reproducibility in LLM-based papers. Nevertheless, there is no work investigating how serious this non-determinism threat is. To fill this gap, this article conducts an empirical study on the non-determinism of ChatGPT in code generation. We chose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems across three code generation benchmarks (i.e., CodeContests, APPS and HumanEval) with three aspects of code similarities: semantic similarity, syntactic similarity, and structural similarity. Our results reveal that ChatGPT exhibits a high degree of non-determinism under the default setting: the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00% and 47.56% for three different code generation datasets (i.e., CodeContests, APPS and HumanEval), respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration ( temperature \(=\) 1). In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.

Funding

  • UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence

Cite

APA

Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025). An Empirical Study of the Non-Determinism of ChatGPT in Code Generation. ACM Transactions on Software Engineering and Methodology, 34(2), 1-28. https://doi.org/10.1145/3697010

Vancouver

Ouyang S, Zhang JM, Harman M, Wang M. An Empirical Study of the Non-Determinism of ChatGPT in Code Generation. ACM Transactions on Software Engineering and Methodology. 2025;34(2):1-28. doi:10.1145/3697010

BibTeX
@article{Ouyang2025,
  author = {Ouyang, Shuyin and Zhang, Jie M. and Harman, Mark and Wang, Meng},
  title = {An Empirical Study of the Non-Determinism of ChatGPT in Code Generation},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2025},
  volume = {34},
  number = {2},
  pages = {1--28},
  doi = {10.1145/3697010}
}
Start for free →