EUVIMEDOdkrijte EUVIMED
Open access · CC BY 4.0

PyDex: Repairing Bugs in Introductory Python Assignments using LLMs

Jialu Zhang, José Pablo Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gustavo Soares, Gust Verbruggen

Proceedings of the ACM on Programming Languages · 2024-04-29 · 8 (OOPSLA1) · 1100-1124

DOI: 10.1145/3649850 ↗

At a glance

Journal article in Proceedings of the ACM on Programming Languages, published online on 2024-04-29. Written by Jialu Zhang and 6 co-authors. Cited 51 times so far (Crossref); 69 references. The full text is freely available (open access, CC BY 4.0). Funded by: NSF.

AbstraktBibliografické údajeCiteRelated studies

Abstract (original language)

Students often make mistakes in their introductory programming assignments as part of their learning process. Unfortunately, providing custom repairs for these mistakes can require a substantial amount of time and effort from class instructors. Automated program repair (APR) techniques can be used to synthesize such fixes. Prior work has explored the use of symbolic and neural techniques for APR in the education domain. Both types of approaches require either substantial engineering efforts or large amounts of data and training. We propose to use a large language model trained on code, such as Codex (a version of GPT), to build an APR system -- PyDex -- for introductory Python programming assignments. Our system can fix both syntactic and semantic mistakes by combining multi-modal prompts, iterative querying, test-case-based selection of few-shots, and program chunking. We evaluate PyDex on 286 real student programs and compare to three baselines, including one that combines a state-of-the-art Python syntax repair engine, BIFI, and a state-of-the-art Python semantic repair engine for student assignments, Refactory. We find that PyDex can fix more programs and produce smaller patches on average.

Funding

  • NSF

Cite

APA

Zhang, J., Cambronero, J. P., Gulwani, S., Le, V., Piskac, R., Soares, G., & Verbruggen, G. (2024). PyDex: Repairing Bugs in Introductory Python Assignments using LLMs. Proceedings of the ACM on Programming Languages, 8(OOPSLA1), 1100-1124. https://doi.org/10.1145/3649850

Vancouver

Zhang J, Cambronero JP, Gulwani S, Le V, Piskac R, Soares G, et al. PyDex: Repairing Bugs in Introductory Python Assignments using LLMs. Proceedings of the ACM on Programming Languages. 2024;8(OOPSLA1):1100-1124. doi:10.1145/3649850

BibTeX
@article{Zhang2024,
  author = {Zhang, Jialu and Cambronero, José Pablo and Gulwani, Sumit and Le, Vu and Piskac, Ruzica and Soares, Gustavo and Verbruggen, Gust},
  title = {PyDex: Repairing Bugs in Introductory Python Assignments using LLMs},
  journal = {Proceedings of the ACM on Programming Languages},
  year = {2024},
  volume = {8},
  number = {OOPSLA1},
  pages = {1100--1124},
  doi = {10.1145/3649850}
}
Start for free →