EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

An Empirical Study of the Non-Determinism of ChatGPT in Code Generation

Shuyin Ouyang, Jie M. Zhang, Mark Harman, Meng Wang

ACM Transactions on Software Engineering and Methodology · 28.02.2025 · 34 (2) · 1-28

DOI: 10.1145/3697010 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 22.01.2025. Verfasst von Shuyin Ouyang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 245-mal zitiert (Crossref) und nennt 73 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; non-deterministically returning very different code for the same prompt. Such non-determinism affects the correctness and consistency of the generated code, undermines developers’ trust in LLMs, and yields low reproducibility in LLM-based papers. Nevertheless, there is no work investigating how serious this non-determinism threat is. To fill this gap, this article conducts an empirical study on the non-determinism of ChatGPT in code generation. We chose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems across three code generation benchmarks (i.e., CodeContests, APPS and HumanEval) with three aspects of code similarities: semantic similarity, syntactic similarity, and structural similarity. Our results reveal that ChatGPT exhibits a high degree of non-determinism under the default setting: the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00% and 47.56% for three different code generation datasets (i.e., CodeContests, APPS and HumanEval), respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration ( temperature \(=\) 1). In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.

Förderung

  • UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence

Zitieren

APA

Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025). An Empirical Study of the Non-Determinism of ChatGPT in Code Generation. ACM Transactions on Software Engineering and Methodology, 34(2), 1-28. https://doi.org/10.1145/3697010

Vancouver

Ouyang S, Zhang JM, Harman M, Wang M. An Empirical Study of the Non-Determinism of ChatGPT in Code Generation. ACM Transactions on Software Engineering and Methodology. 2025;34(2):1-28. doi:10.1145/3697010

BibTeX
@article{Ouyang2025,
  author = {Ouyang, Shuyin and Zhang, Jie M. and Harman, Mark and Wang, Meng},
  title = {An Empirical Study of the Non-Determinism of ChatGPT in Code Generation},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2025},
  volume = {34},
  number = {2},
  pages = {1--28},
  doi = {10.1145/3697010}
}
Kostenlos starten →