Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation
ACM Transactions on Software Engineering and Methodology · 02.06.2026
DOI: 10.1145/3819239 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 02.06.2026. Verfasst von Mohammad Abdollahi und 5 weiteren Autor:innen. Die Arbeit nennt 38 Literaturangaben.
Abstract (Originalsprache)
Understanding a program’s runtime reasoning behavior, i.e., how intermediate states and control flows lead to final execution results, is essential for reliable code generation, debugging, and automated reasoning. While large language models (LLMs) have demonstrated impressive capabilities in predicting program outputs, most prior studies have focused on output accuracy and performance comparisons, treating reasoning as a black box. As a result, models often struggle with complex control logic or subtle semantic nuances, and little is known about the structure, quality, or failure modes of their reasoning traces. Recent advances in reasoning LLMs, which explicitly generate intermediate reasoning steps before producing final answers, raise the expectation that models could not only produce correct code but also explain their reasoning in ways consistent with program semantics. This ability is particularly critical for tasks involving runtime behavior, where reasoning must capture both control flow and state changes. However, the lack of systematic evaluation of reasoning traces leaves open fundamental questions about how reasoning errors arise and how they influence execution correctness. To address this gap, we conduct the first empirical study on reasoning runtime behavior inference with reasoning LLMs to systematically uncover and characterize the errors in their reasoning traces. For our research, we curate an augmented benchmark constructed from three widely used datasets (i.e., HumanEval+, LiveCodeBench, and REPOEXEC), containing 477 code snippets. For each code snippet, we experiment with three different types of inputs, i.e., regular, edge, and invalid inputs. A total of 12 input values are selected per snippet, each paired with its ground-truth execution outcome. We evaluate four state-of-the-art reasoning LLMs, i.e., DeepSeek-R1, OpenAI o4-mini, Gemini 2.5 Flash, and Claude 4 Sonnet, and our experiment results show that the examined reasoning LLMs can achieve high accuracies ranging from 62% to 98% across all three input types. Additionally, we analyzed the errors in reasoning traces produced by LLMs and developed a comprehensive taxonomy that groups nine specific categories of inference errors under four high-level parent categories for code output inference tasks. We further investigate whether a tool-augmented reasoning approach can enhance the reasoning ability of LLMs. Using failures in the Computation Errors category as a case study, our experiments show that this method successfully corrects 69% of such errors, underscoring the promise of tool augmentation in improving LLM reasoning.
Zitieren
APA
Abdollahi, M., Tasnia, K. R., Saha, S. K., Yang, J., Wang, S., & Hemmati, H. (2026). Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3819239
Vancouver
Abdollahi M, Tasnia KR, Saha SK, Yang J, Wang S, Hemmati H. Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation. ACM Transactions on Software Engineering and Methodology. 2026. doi:10.1145/3819239
BibTeX
@article{Abdollahi2026,
author = {Abdollahi, Mohammad and Tasnia, Khandaker Rifah and Saha, Soumit Kanti and Yang, Jinqiu and Wang, Song and Hemmati, Hadi},
title = {Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation},
journal = {ACM Transactions on Software Engineering and Methodology},
year = {2026},
doi = {10.1145/3819239}
}
EUVIMED