Ambiguity-Aware HDFS Log Anomaly Detection with Retrieval-Augmented Failure Narratives and Selective Refusal
Journal of Advanced Computing Systems · 14.01.2023 · 3 (1) · 66-80
DOI: 10.69987/jacs.2023.30105 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Advanced Computing Systems, online erschienen am 14.01.2023. Verfasst von Jiayi Nie und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
This paper studies block-level anomaly detection for Hadoop Distributed File System (HDFS) system logs. The objective is to determine whether anomaly detection, operator-facing explanation, and selective abstention can be integrated into a single reproducible pipeline for highly imbalanced system traces. A reproducible hybrid detector that combines linear discriminative scoring, pattern-memory posteriors, trace statistics, and a calibrated stacking stage was implemented. The evaluation uses the full HDFS_v1 benchmark, which contains 11,175,629 log lines grouped into 575,061 labeled block traces. Retrieval-augmented generation (RAG) is used in a deterministic sense: the system retrieves matched training patterns and renders fixed failure narratives from event templates rather than invoking a free-form large language model. On the chronological test split, the proposed ambiguity-aware stacked detector achieved 0.9881 precision, 0.9869 recall, 0.9875 F1, 0.9975 area under the precision-recall curve (PR-AUC), and 0.99995 area under the receiver operating characteristic curve (ROC-AUC). Simple logistic regression and linear support vector machine (SVM) baselines reached a higher F1 of 0.9952, which indicates that HDFS event counts remain a very strong supervised signal. However, the proposed pipeline produced stronger ranking metrics than those two linear baselines and, at 99.5% automatic coverage, selective refusal increased accepted-case F1 to 0.9980 while removing all accepted false positives. The additive contribution is therefore not a new classifier alone, but an ambiguity-aware framework that combines detection, deterministic RAG-style failure narratives, and refusal decisions. This capability is relevant to petroleum digital operations because distributed storage and data-platform logs increasingly support drilling analytics, seismic processing, production monitoring, and refinery data systems where reliable triage and reviewable incident narratives are needed.
Zitieren
APA
Nie, J., & Zheng, D. (2023). Ambiguity-Aware HDFS Log Anomaly Detection with Retrieval-Augmented Failure Narratives and Selective Refusal. Journal of Advanced Computing Systems, 3(1), 66-80. https://doi.org/10.69987/jacs.2023.30105
Vancouver
Nie J, Zheng D. Ambiguity-Aware HDFS Log Anomaly Detection with Retrieval-Augmented Failure Narratives and Selective Refusal. Journal of Advanced Computing Systems. 2023;3(1):66-80. doi:10.69987/jacs.2023.30105
BibTeX
@article{Nie2023,
author = {Nie, Jiayi and Zheng, David},
title = {Ambiguity-Aware HDFS Log Anomaly Detection with Retrieval-Augmented Failure Narratives and Selective Refusal},
journal = {Journal of Advanced Computing Systems},
year = {2023},
volume = {3},
number = {1},
pages = {66--80},
doi = {10.69987/jacs.2023.30105}
}
EUVIMED