TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models
ACM Transactions on Software Engineering and Methodology · 31.07.2026 · 35 (7) · 1-36
DOI: 10.1145/3773285 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 11.07.2026. Verfasst von Florian Tambon und 4 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 95 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Fonds de Recherche du Québec (FRQ), the Canadian Institute for Advanced Research, Natural Sciences and Engineering Research Council of Canada, Consortium for Research and Innovation in Aerospace in Québec.
Abstract (Originalsprache)
Large Language Models (LLMs) excel in code-related tasks like code generation, but benchmark evaluations often overlook task characteristics such as difficulty. Moreover, benchmarks are usually built using tasks described with a single prompt, despite the formulation of prompts having a profound impact on the outcome. This article introduces a generalist approach, TaskEval, a framework using diverse prompts and Item Response Theory (IRT) to efficiently assess LLMs’ capabilities and benchmark task characteristics, improving the understanding of their performance. Using two code generation benchmarks, HumanEval + and ClassEval , as well as eight code generation LLMs, we show that TaskEval is capable of characterising the properties of tasks. Using topic analysis, we identify and analyse the tasks of 17 and 21 topics within the benchmarks. We also cross-analyse tasks’ characteristics with programming constructs (e.g. variable assignment, conditions) used by LLMs, emphasising some patterns with tasks’ difficulty. Finally, we conduct a comparison between the difficulty assessment of tasks by human annotators and LLMs. Orthogonal to current benchmarking evaluation efforts, TaskEval can assist researchers and practitioners in fostering better assessments of LLMs. The tasks’ characteristics can be used to identify shortcomings within existing benchmarks or improve the evaluation of LLMs.
Förderung
- Fonds de Recherche du Québec (FRQ), the Canadian Institute for Advanced Research
- Natural Sciences and Engineering Research Council of Canada
- Consortium for Research and Innovation in Aerospace in Québec
- Thales Canada Inc., Bell Textron Canada Limited, CAE Inc., and Bombardier Inc.
Zitieren
APA
Tambon, F., Nikanjam, A., Zid, C., Khomh, F., & Antoniol, G. (2026). TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models. ACM Transactions on Software Engineering and Methodology, 35(7), 1-36. https://doi.org/10.1145/3773285
Vancouver
Tambon F, Nikanjam A, Zid C, Khomh F, Antoniol G. TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models. ACM Transactions on Software Engineering and Methodology. 2026;35(7):1-36. doi:10.1145/3773285
BibTeX
@article{Tambon2026,
author = {Tambon, Florian and Nikanjam, Amin and Zid, Cyrine and Khomh, Foutse and Antoniol, Giuliano},
title = {TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models},
journal = {ACM Transactions on Software Engineering and Methodology},
year = {2026},
volume = {35},
number = {7},
pages = {1--36},
doi = {10.1145/3773285}
}
EUVIMED