EUVIMEDEUVIMED entdecken
Crossref

An Empirical Study of Testing Machine Learning in the Wild

Moses Openja, Foutse Khomh, Armstrong Foundjem, Zhen Ming (Jack) Jiang, Mouna Abidi, Ahmed E. Hassan

ACM Transactions on Software Engineering and Methodology · 31.01.2025 · 34 (1) · 1-63

DOI: 10.1145/3680463 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 26.12.2024. Verfasst von Moses Openja und 5 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 151 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Recently, machine and deep learning (ML/DL) algorithms have been increasingly adopted in many software systems. Due to their inductive nature, ensuring the quality of these systems remains a significant challenge for the research community. Traditionally, software systems were constructed deductively, by writing explicit rules that govern the behavior of the system as program code. However, ML/DL systems infer rules from training data i.e., they are generated inductively. Recent research in ML/DL quality assurance has adapted concepts from traditional software testing, such as mutation testing, to improve reliability. However, it is unclear if these proposed testing techniques are adopted in practice, or if new testing strategies have emerged from real-world ML deployments. There is little empirical evidence about the testing strategies.

Aims

To fill this gap, we perform the first fine-grained empirical study on ML testing in the wild to identify the ML properties being tested, the testing strategies, and their implementation throughout the ML workflow.

Method

We conducted a mixed-methods study to understand ML software testing practices. We analyzed test files and cases from 11 open-source ML/DL projects on GitHub. Using open coding, we manually examined the testing strategies, tested ML properties, and implemented testing methods to understand their practical application in building and releasing ML/DL software systems.

Results

Our findings reveal several key insights: (1) The most common testing strategies, accounting for less than 40%, are Grey-box and White-box methods, such as Negative Testing , Oracle Approximation , and Statistical Testing . (2) A wide range of \(17\) ML properties are tested, out of which only 20% to 30% are frequently tested, including Consistency , Correctness , and Efficiency . (3) Bias and Fairness is more tested in Recommendation (6%) and Computer Vision (CV) (3.9%) systems, while Security and Privacy is tested in CV (2%), Application Platforms (0.9%), and NLP (0.5%). (4) We identified 13 types of testing methods, such as Unit Testing , Input Testing , and Model Testing .

Conclusions

This study sheds light on the current adoption of software testing techniques and highlights gaps and limitations in existing ML testing practices.

Zitieren

APA

Openja, M., Khomh, F., Foundjem, A., Jiang, Z. M. J., Abidi, M., & Hassan, A. E. (2025). An Empirical Study of Testing Machine Learning in the Wild. ACM Transactions on Software Engineering and Methodology, 34(1), 1-63. https://doi.org/10.1145/3680463

Vancouver

Openja M, Khomh F, Foundjem A, Jiang ZMJ, Abidi M, Hassan AE. An Empirical Study of Testing Machine Learning in the Wild. ACM Transactions on Software Engineering and Methodology. 2025;34(1):1-63. doi:10.1145/3680463

BibTeX
@article{Openja2025,
  author = {Openja, Moses and Khomh, Foutse and Foundjem, Armstrong and Jiang, Zhen Ming (Jack) and Abidi, Mouna and Hassan, Ahmed E.},
  title = {An Empirical Study of Testing Machine Learning in the Wild},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2025},
  volume = {34},
  number = {1},
  pages = {1--63},
  doi = {10.1145/3680463}
}
Kostenlos starten →