EUVIMEDEUVIMED entdecken
Crossref

Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical Analysis

Roman Haas, Raphael Nömmer, Elmar Juergens, Sven Apel

IEEE Transactions on Software Engineering · 2024-08 · 50 (8) · 2005-2020

DOI: 10.1109/tse.2024.3418191 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Software Engineering, 2024-08. Verfasst von Roman Haas und 3 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 65 Literaturangaben. Gefördert von: German Federal Ministry of Education and Research, German Research Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Context

Both automated and manual software testing are widely applied in practice. While being essential for project success and software quality, they are very resource-intensive, thus motivating the pursuit for optimization. Goal: We aim at understanding to what extent test optimization techniques for automated testing from the field of test case selection, prioritization, and test suite minimization can be applied to manual testing processes in practice.

Method

We have studied the automated and manual testing process of five industrial study subjects from five different domains with different technological backgrounds and assessed the costs and benefits of test optimization techniques in industrial practice. In particular, we have carried out a cost–benefit analysis of two language-agnostic optimization techniques (test impact analysis and Pareto testing a technique we introduce in this paper) on 2,622 real-world failures from our subject's histories.

Results

Both techniques maintain most of the fault detection capability while significantly reducing the test runtime. For automated testing, optimized test suites detect, on average, 80% of failures, while saving 66% of execution time, as compared to 81% failure detection rate for manual test suites and an average time saving of 43%. We observe an average speedup of the time to first failure of around 49 compared to a random test ordering.

Conclusion

Our results suggest that optimization techniques from automated testing can be transferred to manual testing in industrial practice, resulting in lower test execution time and much lower time-to-feedback, but coming with process-related limitations and requirements for a successful implementation. All study subjects implemented one of our test optimization techniques in their processes, which demonstrates the practical impact of our findings.

Förderung

  • German Federal Ministry of Education and Research
  • German Research Foundation

Zitieren

APA

Haas, R., Nömmer, R., Juergens, E., & Apel, S. (2024). Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical Analysis. IEEE Transactions on Software Engineering, 50(8), 2005-2020. https://doi.org/10.1109/tse.2024.3418191

Vancouver

Haas R, Nömmer R, Juergens E, Apel S. Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical Analysis. IEEE Transactions on Software Engineering. 2024;50(8):2005-2020. doi:10.1109/tse.2024.3418191

BibTeX
@article{Haas2024,
  author = {Haas, Roman and Nömmer, Raphael and Juergens, Elmar and Apel, Sven},
  title = {Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical Analysis},
  journal = {IEEE Transactions on Software Engineering},
  year = {2024},
  volume = {50},
  number = {8},
  pages = {2005--2020},
  doi = {10.1109/tse.2024.3418191}
}
Kostenlos starten →