EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models

Yanlin Wang, Suiquan Wang, Yanli Wang, Bowen Zhang, Daya Guo, Jiachi Chen, Zibin Zheng

Proceedings of the ACM on Software Engineering · 30.06.2026 · 3 (FSE) · 2790-2812

DOI: 10.1145/3808131 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 30.06.2026. Verfasst von Yanlin Wang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 63 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent large language models (LLMs) have shown strong performance on software engineering tasks, yet most existing benchmarks evaluate code reasoning at the function level, where all relevant information is localized. This setting fails to reflect real-world development, which requires reasoning across multiple files and complex dependency structures. We introduce RepoReasoner, a benchmark for evaluating repository-level code reasoning. It assesses two complementary abilities: Output Prediction, which measures fine-grained, stateful execution reasoning across files, and Call Chain Prediction, which evaluates high-level architectural dependency understanding under noisy context. Our benchmark is constructed through a multi-stage pipeline that leverages dynamic tracing of pytest executions to obtain ground-truth call chains, along with LLM-based I/O rewriting to reduce memorization effects. We evaluate seven state-of-the-art LLMs. Even under oracle context, the best-performing model achieves only 69.1% Pass@1 on Output Prediction, indicating that cross-file reasoning remains a major challenge. In Call Chain Prediction, models exhibit high precision but low recall, suggesting limited multi-hop dependency understanding. Furthermore, performance drops on rewritten data reveal partial reliance on memorization, and longer contexts do not consistently improve results due to noise. These findings highlight fundamental limitations in current LLMs’ repository-level reasoning and motivate future work on structured architectural understanding and cross-file inference.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Wang, Y., Wang, S., Wang, Y., Zhang, B., Guo, D., Chen, J., & Zheng, Z. (2026). RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models. Proceedings of the ACM on Software Engineering, 3(FSE), 2790-2812. https://doi.org/10.1145/3808131

Vancouver

Wang Y, Wang S, Wang Y, Zhang B, Guo D, Chen J, et al. RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models. Proceedings of the ACM on Software Engineering. 2026;3(FSE):2790-2812. doi:10.1145/3808131

BibTeX
@article{Wang2026,
  author = {Wang, Yanlin and Wang, Suiquan and Wang, Yanli and Zhang, Bowen and Guo, Daya and Chen, Jiachi and Zheng, Zibin},
  title = {RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models},
  journal = {Proceedings of the ACM on Software Engineering},
  year = {2026},
  volume = {3},
  number = {FSE},
  pages = {2790--2812},
  doi = {10.1145/3808131}
}
Kostenlos starten →