EUVIMEDEUVIMED entdecken
Crossref

LLMRCA: Multilevel Root Cause Analysis for LLM Applications Using Multimodal Observability Data

Gou Tan, Zilong He, Min Li, Haiyu Huang, Yilun Wang, Pengfei Chen, Giuliano Casale, Chuanfu Zhang

ACM Transactions on Software Engineering and Methodology · 01.04.2026

DOI: 10.1145/3806200 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 01.04.2026. Verfasst von Gou Tan und 7 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 55 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Ensuring the reliability of large language model (LLM) applications in production environments is critical as LLMs are widely integrated into software systems. However, fault diagnosis in these applications is challenging due to distributed deployment and complex component interactions. Existing root cause analysis (RCA) methods fall short when applied to LLM applications because they ignore two key challenges: unstable response times and response quality-related silent faults. To address these challenges, we propose LLMRCA, the first multilevel and unsupervised RCA framework that leverages multimodal observability data to identify root causes in LLM applications. LLMRCA constructs a heterogeneous causal graph from metrics, logs, and traces, and employs a Residual Graph Attention autoencoder to calculate reconstruction errors for RCA. LLMRCA also introduces request classification to handle unstable response times and verification to reduce false positives. Experiments on a retrieval-augmented generation LLM application show that LLMRCA outperforms eight baseline methods, achieving up to \(5.1\times\) more accurate results than baselines for performance problems and 92.86% top-1 ranking accuracy for response-quality problems. Furthermore, the ablation studies confirm the contributions of each data modality and different framework modules to effectiveness. Generally, our framework provides a practical approach for diagnosing both performance and quality faults in LLM applications.

Zitieren

APA

Tan, G., He, Z., Li, M., Huang, H., Wang, Y., Chen, P., Casale, G., & Zhang, C. (2026). LLMRCA: Multilevel Root Cause Analysis for LLM Applications Using Multimodal Observability Data. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3806200

Vancouver

Tan G, He Z, Li M, Huang H, Wang Y, Chen P, et al. LLMRCA: Multilevel Root Cause Analysis for LLM Applications Using Multimodal Observability Data. ACM Transactions on Software Engineering and Methodology. 2026. doi:10.1145/3806200

BibTeX
@article{Tan2026,
  author = {Tan, Gou and He, Zilong and Li, Min and Huang, Haiyu and Wang, Yilun and Chen, Pengfei and Casale, Giuliano and Zhang, Chuanfu},
  title = {LLMRCA: Multilevel Root Cause Analysis for LLM Applications Using Multimodal Observability Data},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2026},
  doi = {10.1145/3806200}
}
Kostenlos starten →