Empirical Insights of Test Selection Metrics under Multiple Testing Objectives and Distribution Shifts
Proceedings of the ACM on Software Engineering · 30.06.2026 · 3 (FSE) · 1288-1311
DOI: 10.1145/3797086 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 30.06.2026. Verfasst von Jingyu Zhang und 5 weiteren Autor:innen. Die Arbeit nennt 83 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Research Grants Council of Hong Kong, JST CRONOS Grant.
Abstract (Originalsprache)
Deep learning (DL)-based systems can exhibit unexpected behavior when exposed to out-of-distribution (OOD) scenarios, posing serious risks in safety-critical domains such as malware detection and autonomous driving. This underscores the importance of thoroughly testing such systems before deployment. To this end, researchers have proposed a wide range of test selection metrics designed to effectively select inputs. However, prior evaluations of metrics reveal three key limitations: (1) narrow testing objectives, for example, many studies assess metrics only for fault detection, leaving their effectiveness for performance estimation unclear; (2) limited coverage of OOD scenarios, with natural and label shifts are rarely considered; (3) Biased dataset selection, where most work focuses on image data while other modalities remain underexplored. Consequently, a unified benchmark that examines how these metrics perform under multiple testing objectives, diverse OOD scenarios, and different data modalities is still lacking. This leaves practitioners uncertain about which test selection metrics are most suitable for their specific objectives and contexts. To address this gap, we conduct an extensive empirical study of 15 existing metrics, evaluating them under three testing objectives (fault detection, performance estimation, and retraining guidance), five types of OOD scenarios (corrupted, adversarial, temporal, natural, and label shifts), three data modalities (image, text, and Android packages), and 13 DL models. In total, our study encompasses 1,640 experimental scenarios, offering a comprehensive evaluation and statistical analysis.
Förderung
- National Natural Science Foundation of China
- Research Grants Council of Hong Kong
- JST CRONOS Grant
- JSPS KAKENHI Grant
- Shenzhen Science and Technology Program
- CityU Funds
Zitieren
APA
Zhang, J., Wang, F., Keung, J., Liao, Y., Xiao, Y., & Ma, L. (2026). Empirical Insights of Test Selection Metrics under Multiple Testing Objectives and Distribution Shifts. Proceedings of the ACM on Software Engineering, 3(FSE), 1288-1311. https://doi.org/10.1145/3797086
Vancouver
Zhang J, Wang F, Keung J, Liao Y, Xiao Y, Ma L. Empirical Insights of Test Selection Metrics under Multiple Testing Objectives and Distribution Shifts. Proceedings of the ACM on Software Engineering. 2026;3(FSE):1288-1311. doi:10.1145/3797086
BibTeX
@article{Zhang2026,
author = {Zhang, Jingyu and Wang, Fan and Keung, Jacky and Liao, Yihan and Xiao, Yan and Ma, Lei},
title = {Empirical Insights of Test Selection Metrics under Multiple Testing Objectives and Distribution Shifts},
journal = {Proceedings of the ACM on Software Engineering},
year = {2026},
volume = {3},
number = {FSE},
pages = {1288--1311},
doi = {10.1145/3797086}
}
EUVIMED