EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?

Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, Shuvendu K. Lahiri

Proceedings of the ACM on Software Engineering · 12.07.2024 · 1 (FSE) · 1889-1912

DOI: 10.1145/3660791 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 12.07.2024. Verfasst von Madeline Endres und 3 weiteren Autor:innen. Die Arbeit wurde bisher 71-mal zitiert (Crossref) und nennt 59 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Informal natural language that describes code functionality, such as code comments or function documentation, may contain substantial information about a program’s intent. However, there is typically no guarantee that a program’s implementation and natural language documentation are aligned. In the case of a conflict, leveraging information in code-adjacent natural language has the potential to enhance fault localization, debugging, and code trustworthiness. In practice, however, this information is often underutilized due to the inherent ambiguity of natural language, which makes natural language intent challenging to check programmatically. The “emergent abilities” of Large Language Models (LLMs) have the potential to facilitate the translation of natural language intent to programmatically checkable assertions. However, it is unclear if LLMs can correctly translate informal natural language specifications into formal specifications that match programmer intent. Additionally, it is unclear if such translation could be useful in practice. In this paper, we describe nl2postcond , the problem of leveraging LLMs for transforming informal natural language to formal method postconditions, expressed as program assertions. We introduce and validate metrics to measure and compare different nl2postcond approaches, using the correctness and discriminative power of generated postconditions. We then use qualitative and quantitative methods to assess the quality of nl2postcond postconditions, finding that they are generally correct and able to discriminate incorrect code. Finally, we find that nl2postcond via LLMs has the potential to be helpful in practice; nl2postcond generated postconditions were able to catch 64 real-world historical bugs from Defects4J .

Zitieren

APA

Endres, M., Fakhoury, S., Chakraborty, S., & Lahiri, S. K. (2024). Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?. Proceedings of the ACM on Software Engineering, 1(FSE), 1889-1912. https://doi.org/10.1145/3660791

Vancouver

Endres M, Fakhoury S, Chakraborty S, Lahiri SK. Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?. Proceedings of the ACM on Software Engineering. 2024;1(FSE):1889-1912. doi:10.1145/3660791

BibTeX
@article{Endres2024,
  author = {Endres, Madeline and Fakhoury, Sarah and Chakraborty, Saikat and Lahiri, Shuvendu K.},
  title = {Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?},
  journal = {Proceedings of the ACM on Software Engineering},
  year = {2024},
  volume = {1},
  number = {FSE},
  pages = {1889--1912},
  doi = {10.1145/3660791}
}
Kostenlos starten →