Understanding the Potentially Confounding Effect of Test Suite Size in Test Effectiveness Evaluation
ACM Transactions on Software Engineering and Methodology · 31.05.2026 · 35 (5) · 1-54
DOI: 10.1145/3748504 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 24.04.2026. Verfasst von Zeyu Lu und 11 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 120 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Jiangsu Natural Science Foundation.
Abstract (Originalsprache)
Background
Code coverage and mutation score serve as pivotal test effectiveness metrics used to assess a test suite’s ability to uncover actual defects. However, prior research has produced inconsistent or even conflicting findings regarding their correlation with defect detection capability, particularly concerning the impact of test suite size. Problem. The extent of the potentially confounding effect of test suite size in test effectiveness evaluation context is not clear, nor is the method to remove the potentially confounding effect, or the influence of this removal on the performance of test suite optimization.
Objective
Our goal is to deeply understand how test suite size affects the true relationship between test effectiveness metrics and a test suite’s ability to detect actual defects.
Method
We first employ statistical methods to examine the extent of the potentially confounding effect of test suite size in the context of test effectiveness evaluation. After that, we propose a linear regression-based method to remove the potentially confounding effect of test suite size. Finally, we empirically explore the impact of this removal method on test suite optimization.
Result
Our experimental results, based on the Defects4J defect dataset, uncovers that: (1) the confounding effect of test suite size on the associations between test effectiveness metrics and defect detection capability in general exists; (2) the proposed linear regression-based method can effectively remove the confounding effect; and (3) after removing the confounding effect, mutation score demonstrates superior effectiveness in predicting test suite effectiveness, while statement coverage is the least effective metric. Furthermore, both coverage-based and mutation-based test suite reduction exhibit enhanced cost-effectiveness in defect detection, and there is a marginal improvement in the speed of defect detection for coverage-based test case prioritization.
Conclusion
When using test effectiveness metrics to assess test suite effectiveness, it is crucial to eliminate the influence of test suite size.
Förderung
- National Natural Science Foundation of China
- Jiangsu Natural Science Foundation
Zitieren
APA
Lu, Z., Wang, Y., Rong, Y., Huang, Y., Wang, X., Zhang, P., Gao, S., Sun, M., Yang, Y., Li, Y., Chen, L., & Zhou, Y. (2026). Understanding the Potentially Confounding Effect of Test Suite Size in Test Effectiveness Evaluation. ACM Transactions on Software Engineering and Methodology, 35(5), 1-54. https://doi.org/10.1145/3748504
Vancouver
Lu Z, Wang Y, Rong Y, Huang Y, Wang X, Zhang P, et al. Understanding the Potentially Confounding Effect of Test Suite Size in Test Effectiveness Evaluation. ACM Transactions on Software Engineering and Methodology. 2026;35(5):1-54. doi:10.1145/3748504
BibTeX
@article{Lu2026,
author = {Lu, Zeyu and Wang, Yang and Rong, Yi and Huang, Yifan and Wang, Xuan and Zhang, Peng and Gao, Shan and Sun, Maolin and Yang, Yibiao and Li, Yanhui and Chen, Lin and Zhou, Yuming},
title = {Understanding the Potentially Confounding Effect of Test Suite Size in Test Effectiveness Evaluation},
journal = {ACM Transactions on Software Engineering and Methodology},
year = {2026},
volume = {35},
number = {5},
pages = {1--54},
doi = {10.1145/3748504}
}
EUVIMED