EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Confidence vs. Competence: Misalignment in Judgment and Performance for Agentic Software Repair

Mingyue Yuan, Jieshan Chen, Dehai Zhao, Gelareh Mohammadi, Yoshifumi Kitamura, Aaron John Quigley, Qinghua Lu, Zhenchang Xing

ACM Transactions on Software Engineering and Methodology · 30.11.2026 · 35 (11) · 1-45

DOI: 10.1145/3822603 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 04.11.2026. Verfasst von Mingyue Yuan und 7 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 54 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Dieter Schwarz Foundation and the Technical University of Munich – Institute for Advanced Study, Germany.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

While Large Language Model (LLM) agents are increasingly applied to automated software repair, misalignment remains in how humans and agents judge issue conditions and how agents’ pre-execution self-assessments relate to repair competence. Human developers rely on diagnostic cues such as reproduction steps and stack traces to judge whether an issue is sufficiently specified, whereas LLM agents often fail to recognize missing information. We present the first systematic empirical study of misalignment between human and agent judgments and between agent judgments and repair performance. Using SWE-bench, controlled ablation experiments establish a causal link between removing human-valued cues and reduced LLM repair success. Specifically, LLM judges show limited agreement with human problem-specification ratings, and agents’ pre-execution self-assessments only weakly track repair degradation when key cues are removed. Our trajectory analysis further reveals distinct behavioral responses to missing information, while post-execution self-judgment signals add useful discriminative information when combined with behavioral traces. These findings reveal a persistent gap between how current repair agents assess issue conditions and how they perform during repair, highlighting the need for judgment-aware and correction-aware agent design.

Förderung

  • Dieter Schwarz Foundation and the Technical University of Munich – Institute for Advanced Study, Germany

Zitieren

APA

Yuan, M., Chen, J., Zhao, D., Mohammadi, G., Kitamura, Y., Quigley, A. J., Lu, Q., & Xing, Z. (2026). Confidence vs. Competence: Misalignment in Judgment and Performance for Agentic Software Repair. ACM Transactions on Software Engineering and Methodology, 35(11), 1-45. https://doi.org/10.1145/3822603

Vancouver

Yuan M, Chen J, Zhao D, Mohammadi G, Kitamura Y, Quigley AJ, et al. Confidence vs. Competence: Misalignment in Judgment and Performance for Agentic Software Repair. ACM Transactions on Software Engineering and Methodology. 2026;35(11):1-45. doi:10.1145/3822603

BibTeX
@article{Yuan2026,
  author = {Yuan, Mingyue and Chen, Jieshan and Zhao, Dehai and Mohammadi, Gelareh and Kitamura, Yoshifumi and Quigley, Aaron John and Lu, Qinghua and Xing, Zhenchang},
  title = {Confidence vs. Competence: Misalignment in Judgment and Performance for Agentic Software Repair},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2026},
  volume = {35},
  number = {11},
  pages = {1--45},
  doi = {10.1145/3822603}
}
Kostenlos starten →