Robust and Transferable Log-based Anomaly Detection
Proceedings of the ACM on Management of Data · 26.05.2023 · 1 (1) · 1-26
DOI: 10.1145/3588918 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Management of Data, online erschienen am 30.05.2023. Verfasst von Peng Jia und 4 weiteren Autor:innen. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 68 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Shenzhen Basic Research Grant, the National Key R&D Program of China, National Research Foundation, Singapore under its Emerging Areas Research Projects (EARP) Funding Initiative..
Abstract (Originalsprache)
Log messages provide a valuable source of runtime information for ensuring the safety and consistency of systems. Recently, many machine learning and deep learning methods have been proposed to automatically detect anomalous log messages, obviating the need for manual detection by experts. However, we find that in practice, the effectiveness of existing learning-based methods is severely affected by incomplete information and distribution shift. Specifically, each log message can actually be parsed into a fixed number of key information fields, while existing methods analyze log messages using only the log event information and ignore other useful information fields that can be critical to anomaly detection. Further, the distribution of real-world log messages changes continuously due to the dynamic nature of the runtime environment and thus, a detection model conventionally trained based on the unrealistic i.i.d. assumption may not provide the expected and consistent performance. In this paper, we present a robust and transferable anomaly detection framework RT-Log to address the above problems. To perform a comprehensive analysis of log messages, we introduce an adaptive relation modeling technique, which captures feature interactions among log information fields selectively and dynamically for effective and interpretable log representations. To establish its robustness and transferability, we propose a general environment generalization technique for learning the environment invariant representations that can generalize across different runtime environments. We evaluate the anomaly detection performance of RT-Log on large real-world datasets. Extensive experimental results demonstrate that RT-Log consistently outperforms state-of-the-art methods by a significant margin under different settings.
Förderung
- Shenzhen Basic Research Grant
- the National Key R&D Program of China
- National Research Foundation, Singapore under its Emerging Areas Research Projects (EARP) Funding Initiative.
- National Natural Science Foundation of China
Zitieren
APA
Jia, P., Cai, S., Ooi, B. C., Wang, P., & Xiong, Y. (2023). Robust and Transferable Log-based Anomaly Detection. Proceedings of the ACM on Management of Data, 1(1), 1-26. https://doi.org/10.1145/3588918
Vancouver
Jia P, Cai S, Ooi BC, Wang P, Xiong Y. Robust and Transferable Log-based Anomaly Detection. Proceedings of the ACM on Management of Data. 2023;1(1):1-26. doi:10.1145/3588918
BibTeX
@article{Jia2023,
author = {Jia, Peng and Cai, Shaofeng and Ooi, Beng Chin and Wang, Pinghui and Xiong, Yiyuan},
title = {Robust and Transferable Log-based Anomaly Detection},
journal = {Proceedings of the ACM on Management of Data},
year = {2023},
volume = {1},
number = {1},
pages = {1--26},
doi = {10.1145/3588918}
}
EUVIMED