Bias Testing and Mitigation in LLM-based Code Generation
ACM Transactions on Software Engineering and Methodology · 31.01.2026 · 35 (1) · 1-31
DOI: 10.1145/3724117 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 11.12.2025. Verfasst von Dong Huang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 40-mal zitiert (Crossref) und nennt 130 Literaturangaben. Gefördert von: National Key R&D Program of China, HK RGC RIF, Huawei Flagship Research.
Abstract (Originalsprache)
As the adoption of LLMs becomes more widespread in software coding ecosystems, a pressing issue has emerged: does the generated code contain social bias and unfairness, such as those related to age, gender, and race? This issue concerns the integrity, fairness, and ethical foundation of software applications that depend on the code generated by these models but are underexplored in the literature. This article presents a novel bias testing framework that is specifically designed for code generation tasks. Based on this framework, we conduct an extensive empirical study on the biases in code generated by five widely studied LLMs (i.e., PALM-2-CodeChat-bison, Claude-instant-1, GPT-3.5-turbo, GPT-4-turbo, and GPT-4). Our findings reveal that biases are prevalent. For example, 13.47% to 49.10% of the codes generated by these LLMs have biased behaviors towards gender. Moreover, we study five bias mitigation prompt strategies that are commonly used in current code generation scenarios, i.e., zero-shot, one-shot, few-shot, and two Chain-of-Thought (CoT) prompts, with and without provided feedback-driven refinement. Our evaluation results illustrate that using direct prompt engineering strategies has limited effectiveness in mitigating bias, but our test execution feedback can help to reduce the ratio of code biases to a large extent (e.g., from 59.88% to 4.79% for GPT-4).
Förderung
- National Key R&D Program of China
- HK RGC RIF
- Huawei Flagship Research
- HK RGC GRF
- HKU-CAS Joint Laboratory for Intelligent System Software
Zitieren
APA
Huang, D., Zhang, J. M., Bu, Q., Xie, X., Chen, J., & Cui, H. (2026). Bias Testing and Mitigation in LLM-based Code Generation. ACM Transactions on Software Engineering and Methodology, 35(1), 1-31. https://doi.org/10.1145/3724117
Vancouver
Huang D, Zhang JM, Bu Q, Xie X, Chen J, Cui H. Bias Testing and Mitigation in LLM-based Code Generation. ACM Transactions on Software Engineering and Methodology. 2026;35(1):1-31. doi:10.1145/3724117
BibTeX
@article{Huang2026,
author = {Huang, Dong and Zhang, Jie M. and Bu, Qingwen and Xie, Xiaofei and Chen, Junjie and Cui, Heming},
title = {Bias Testing and Mitigation in LLM-based Code Generation},
journal = {ACM Transactions on Software Engineering and Methodology},
year = {2026},
volume = {35},
number = {1},
pages = {1--31},
doi = {10.1145/3724117}
}
EUVIMED