Traceable Metamorphic Test Cases for Robust Safety‐Critical Systems: A Deep Learning LiDAR Object Detector Example
Software Testing, Verification and Reliability · 2026-05 · 36 (3-4)
DOI: 10.1002/stvr.70020 ↗Kurz gefasst
Fachartikel in der Zeitschrift Software Testing, Verification and Reliability, online erschienen am 23.04.2026. Verfasst von Simon Speth und 3 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 64 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Assessing the robustness of safety‐critical deep learning (DL) systems is of utmost importance, as these systems can cause harm when deployed in the real world. Metamorphic testing (MT) is one commonly used method to evaluate the robustness of DL systems, as it does not require expensive labelled ground truth data. This paper tackles two challenges: (1) One challenge in regulated domains such as the automotive industry is to provide a traceable argumentation of why a certain metamorphic relation (MR) was chosen. We adopt the idea of defect‐based testing to MT and argue that an MR is traceable if it can be linked to a defect hypothesis. We demonstrate how to assess the robustness of safety‐critical DL systems using the example of LiDAR object detectors. To this end, we create three new MRs for the LiDAR domain and identify five MR that can be reused by adapting them from related domains. Our experiments on the nuScenes dataset with three different object detectors produce 3.9 million test verdicts, of which 0.7 million are test failures. This shows that our defect‐based MR effectively uncover failures. (2) A second challenge resulting from executing numerous metamorphic test cases is that MT can lead to the generation of an impractically high number of failures. We show how to prioritize the most critical failures, such as failures that occur close to the ego vehicle. By prioritizing, we reduced the observed 685,000 failures to 5397 safety‐critical failures corresponding to a 127‐fold reduction.
Zitieren
APA
Speth, S., Springer, T., Fischbach, J., & Pretschner, A. (2026). Traceable Metamorphic Test Cases for Robust Safety‐Critical Systems: A Deep Learning LiDAR Object Detector Example. Software Testing, Verification and Reliability, 36(3-4). https://doi.org/10.1002/stvr.70020
Vancouver
Speth S, Springer T, Fischbach J, Pretschner A. Traceable Metamorphic Test Cases for Robust Safety‐Critical Systems: A Deep Learning LiDAR Object Detector Example. Software Testing, Verification and Reliability. 2026;36(3-4). doi:10.1002/stvr.70020
BibTeX
@article{Speth2026,
author = {Speth, Simon and Springer, Tobias and Fischbach, Jannik and Pretschner, Alexander},
title = {Traceable Metamorphic Test Cases for Robust Safety‐Critical Systems: A Deep Learning LiDAR Object Detector Example},
journal = {Software Testing, Verification and Reliability},
year = {2026},
volume = {36},
number = {3-4},
doi = {10.1002/stvr.70020}
}
EUVIMED