EUVIMEDEUVIMED entdecken
Crossref

Can Large Language Models Reason About Complex Execution Paths? An Empirical Study on Python

Wenhan Wang, Kaibo Liu, Zeyu Sun, An Ran Chen, Ge Li, Gang Huang, Lei Ma

ACM Transactions on Software Engineering and Methodology · 26.06.2026

DOI: 10.1145/3822511 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 26.06.2026. Verfasst von Wenhan Wang und 6 weiteren Autor:innen. Die Arbeit nennt 51 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Execution path reasoning is a key step towards program semantics understanding. It is crucial for generating test cases that cover certain branches/paths, or detecting bugs that are triggered by some paths without actually executing the program. Traditionally, execution path reasoning can be achieved by symbolic execution techniques, but existing SMT-based symbolic execution approaches struggle with complex data structures and external API calls. This challenge is even more pronounced in languages with highly flexible syntax, such as Python, resulting in a lack of widely adopted tools for reasoning on execution paths. Therefore, reasoning execution paths with AI-based approaches become a promising direction. In this paper, we investigate the feasibility of adopting large language models (LLMs) for execution path reasoning on Python, where traditional path-based symbolic execution tools are unavailable. We conduct an empirical study on two types of path reasoning tasks: generation tasks for test case generation and classification tasks for bug detection. We build new evaluation pipelines and benchmarks from both competition-level programs and real-world repositories. Our results show that state-of-theart LLMs can perform correct reasoning on execution paths and improve test coverage on real-world software, though models with stronger reasoning abilities do not always outperform weaker ones. These findings highlight the potential of utilizing LLMs as a complementary heuristic for path-aware code reasoning, especially in program languages lacking mature symbolic execution tools. We have released our benchmark and evaluation scripts at https://github.com/jacobwwh/llm-path-study .

Zitieren

APA

Wang, W., Liu, K., Sun, Z., Chen, A. R., Li, G., Huang, G., & Ma, L. (2026). Can Large Language Models Reason About Complex Execution Paths? An Empirical Study on Python. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3822511

Vancouver

Wang W, Liu K, Sun Z, Chen AR, Li G, Huang G, et al. Can Large Language Models Reason About Complex Execution Paths? An Empirical Study on Python. ACM Transactions on Software Engineering and Methodology. 2026. doi:10.1145/3822511

BibTeX
@article{Wang2026,
  author = {Wang, Wenhan and Liu, Kaibo and Sun, Zeyu and Chen, An Ran and Li, Ge and Huang, Gang and Ma, Lei},
  title = {Can Large Language Models Reason About Complex Execution Paths? An Empirical Study on Python},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2026},
  doi = {10.1145/3822511}
}
Kostenlos starten →