EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Ambiguity-Aware HDFS Log Anomaly Detection with Retrieval-Augmented Failure Narratives and Selective Refusal

Jiayi Nie, David Zheng

Journal of Advanced Computing Systems · 14.01.2023 · 3 (1) · 66-80

DOI: 10.69987/jacs.2023.30105 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Advanced Computing Systems, online erschienen am 14.01.2023. Verfasst von Jiayi Nie und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper studies block-level anomaly detection for Hadoop Distributed File System (HDFS) system logs. The objective is to determine whether anomaly detection, operator-facing explanation, and selective abstention can be integrated into a single reproducible pipeline for highly imbalanced system traces. A reproducible hybrid detector that combines linear discriminative scoring, pattern-memory posteriors, trace statistics, and a calibrated stacking stage was implemented. The evaluation uses the full HDFS_v1 benchmark, which contains 11,175,629 log lines grouped into 575,061 labeled block traces. Retrieval-augmented generation (RAG) is used in a deterministic sense: the system retrieves matched training patterns and renders fixed failure narratives from event templates rather than invoking a free-form large language model. On the chronological test split, the proposed ambiguity-aware stacked detector achieved 0.9881 precision, 0.9869 recall, 0.9875 F1, 0.9975 area under the precision-recall curve (PR-AUC), and 0.99995 area under the receiver operating characteristic curve (ROC-AUC). Simple logistic regression and linear support vector machine (SVM) baselines reached a higher F1 of 0.9952, which indicates that HDFS event counts remain a very strong supervised signal. However, the proposed pipeline produced stronger ranking metrics than those two linear baselines and, at 99.5% automatic coverage, selective refusal increased accepted-case F1 to 0.9980 while removing all accepted false positives. The additive contribution is therefore not a new classifier alone, but an ambiguity-aware framework that combines detection, deterministic RAG-style failure narratives, and refusal decisions. This capability is relevant to petroleum digital operations because distributed storage and data-platform logs increasingly support drilling analytics, seismic processing, production monitoring, and refinery data systems where reliable triage and reviewable incident narratives are needed.

Zitieren

APA

Nie, J., & Zheng, D. (2023). Ambiguity-Aware HDFS Log Anomaly Detection with Retrieval-Augmented Failure Narratives and Selective Refusal. Journal of Advanced Computing Systems, 3(1), 66-80. https://doi.org/10.69987/jacs.2023.30105

Vancouver

Nie J, Zheng D. Ambiguity-Aware HDFS Log Anomaly Detection with Retrieval-Augmented Failure Narratives and Selective Refusal. Journal of Advanced Computing Systems. 2023;3(1):66-80. doi:10.69987/jacs.2023.30105

BibTeX
@article{Nie2023,
  author = {Nie, Jiayi and Zheng, David},
  title = {Ambiguity-Aware HDFS Log Anomaly Detection with Retrieval-Augmented Failure Narratives and Selective Refusal},
  journal = {Journal of Advanced Computing Systems},
  year = {2023},
  volume = {3},
  number = {1},
  pages = {66--80},
  doi = {10.69987/jacs.2023.30105}
}
Kostenlos starten →