EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

L2CEval : Evaluating Language-to-Code Generation Capabilities of Large Language Models

Ansong Ni, Pengcheng Yin, Yilun Zhao, Martin Riddell, Troy Feng, Rui Shen, Stephen Yin, Ye Liu, Semih Yavuz, Caiming Xiong, Shafiq Joty, Yingbo Zhou und 3 weitere

Transactions of the Association for Computational Linguistics · 02.10.2024 · 12 · 1311-1329

DOI: 10.1162/tacl_a_00705 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 02.10.2024. Verfasst von Ansong Ni und 14 weiteren Autor:innen. Die Arbeit wurde bisher 25-mal zitiert (Crossref) und nennt 83 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recently, large language models (LLMs), especially those that are pretrained on code, have demonstrated strong capabilities in generating programs from natural language inputs. Despite promising results, there is a notable lack of a comprehensive evaluation of these models’ language-to-code generation capabilities. Existing studies often focus on specific tasks, model architectures, or learning paradigms, leading to a fragmented understanding of the overall landscape. In this work, we present L2CEval, a systematic evaluation of the language-to-code generation capabilities of LLMs on 7 tasks across the domain spectrum of semantic parsing, math reasoning, and Python programming, analyzing the factors that potentially affect their performance, such as model size, pretraining data, instruction tuning, and different prompting methods. In addition, we assess confidence calibration, and conduct human evaluations to identify typical failures across different tasks and models. L2CEval offers a comprehensive understanding of the capabilities and limitations of LLMs in language-to-code generation. We release the evaluation framework1 and all model outputs, hoping to lay the groundwork for further future research. All future evaluations (e.g., LLaMA-3, StarCoder2, etc) will be updated on the project website: https://l2c-eval.github.io/.

Zitieren

APA

Ni, A., Yin, P., Zhao, Y., Riddell, M., Feng, T., Shen, R., Yin, S., Liu, Y., Yavuz, S., Xiong, C., Joty, S., Zhou, Y., Radev, D., Cohan, A., & Cohan, A. (2024). L2CEval : Evaluating Language-to-Code Generation Capabilities of Large Language Models. Transactions of the Association for Computational Linguistics, 12, 1311-1329. https://doi.org/10.1162/tacl_a_00705

Vancouver

Ni A, Yin P, Zhao Y, Riddell M, Feng T, Shen R, et al. L2CEval : Evaluating Language-to-Code Generation Capabilities of Large Language Models. Transactions of the Association for Computational Linguistics. 2024;12:1311-1329. doi:10.1162/tacl_a_00705

BibTeX
@article{Ni2024,
  author = {Ni, Ansong and Yin, Pengcheng and Zhao, Yilun and Riddell, Martin and Feng, Troy and Shen, Rui and Yin, Stephen and Liu, Ye and Yavuz, Semih and Xiong, Caiming and Joty, Shafiq and Zhou, Yingbo and Radev, Dragomir and Cohan, Arman and Cohan, Arman},
  title = {L2CEval : Evaluating Language-to-Code Generation Capabilities of Large Language Models},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  volume = {12},
  pages = {1311--1329},
  doi = {10.1162/tacl_a_00705}
}
Kostenlos starten →