EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction

Fatemeh Hadadi, Joshua H. Dawes, Donghwan Shin, Domenico Bianculli, Lionel Briand

Empirical Software Engineering · 2024-09 · 29 (5)

DOI: 10.1007/s10664-024-10501-4 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Empirical Software Engineering, online erschienen am 20.06.2024. Verfasst von Fatemeh Hadadi und 4 weiteren Autor:innen. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 81 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Canada Research Chairs, HORIZON EUROPE Framework Programme.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the increasing complexity and scope of software systems, their dependability is crucial. The analysis of log data recorded during system execution can enable engineers to automatically predict failures at run time. Several Machine Learning (ML) techniques, including traditional ML and Deep Learning (DL), have been proposed to automate such tasks. However, current empirical studies are limited in terms of covering all main DL types—Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), and transformer—as well as examining them on a wide range of diverse datasets. In this paper, we aim to address these issues by systematically investigating the combination of log data embedding strategies and DL types for failure prediction. To that end, we propose a modular architecture to accommodate various configurations of embedding strategies and DL-based encoders. To further investigate how dataset characteristics such as dataset size and failure percentage affect model accuracy, we synthesised 360 datasets, with varying characteristics, for three distinct system behavioural models, based on a systematic and automated generation approach. Using the F1 score metric, our results show that the best overall performing configuration is a CNN-based encoder with Logkey2vec. Additionally, we provide specific dataset conditions, namely a dataset size $$>350$$ > 350 or a failure percentage $$>7.5\%$$ > 7.5 % , under which this configuration demonstrates high accuracy for failure prediction.

Förderung

  • Canada Research Chairs
  • HORIZON EUROPE Framework Programme

Zitieren

APA

Hadadi, F., Dawes, J. H., Shin, D., Bianculli, D., & Briand, L. (2024). Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction. Empirical Software Engineering, 29(5). https://doi.org/10.1007/s10664-024-10501-4

Vancouver

Hadadi F, Dawes JH, Shin D, Bianculli D, Briand L. Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction. Empirical Software Engineering. 2024;29(5). doi:10.1007/s10664-024-10501-4

BibTeX
@article{Hadadi2024,
  author = {Hadadi, Fatemeh and Dawes, Joshua H. and Shin, Donghwan and Bianculli, Domenico and Briand, Lionel},
  title = {Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction},
  journal = {Empirical Software Engineering},
  year = {2024},
  volume = {29},
  number = {5},
  doi = {10.1007/s10664-024-10501-4}
}
Kostenlos starten →