Walk the Talk: Is Your Log-based Software Reliability Maintenance System Really Reliable?

2025-09-29

AI Summary

Existing log-based anomaly detection models are predominantly black-box, lacking interpretability—hindering their trustworthy deployment in production environments. Method: This paper introduces “diagnostic fidelity” as a novel metric to quantify explanation credibility, and proposes FaithLog—a causal-guided attention mechanism coupled with adversarial consistency learning. FaithLog jointly models log sequences and evaluates event-level perturbations to ensure verifiable root-cause localization and alignment between explanations and predictions. Contribution/Results: Experiments on two public benchmarks and one industrial dataset demonstrate that FaithLog significantly improves both explanation fidelity and detection accuracy. Its diagnostic fidelity achieves state-of-the-art performance, marking the first systematic effort to identify and bridge the gap between interpretability and reliability in log analysis.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Web traffic and log analysis
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Log-based software reliability maintenance systems are crucial for sustaining stable customer experience. However, existing deep learning-based methods represent a black box for service providers, making it impossible for providers to understand how these methods detect anomalies, thereby hindering trust and deployment in real production environments. To address this issue, this paper defines a trustworthiness metric, diagnostic faithfulness, for models to gain service providers' trust, based on surveys of SREs at a major cloud provider. We design two evaluation tasks: attention-based root cause localization and event perturbation. Empirical studies demonstrate that existing methods perform poorly in diagnostic faithfulness. Consequently, we propose FaithLog, a faithful log-based anomaly detection system, which achieves faithfulness through a carefully designed causality-guided attention mechanism and adversarial consistency learning. Evaluation results on two public datasets and one industrial dataset demonstrate that the proposed method achieves state-of-the-art performance in diagnostic faithfulness.

Problem

• Evaluating trustworthiness of log-based anomaly detection systems for reliability maintenance
• Addressing black-box limitations in deep learning methods for log analysis
• Improving diagnostic faithfulness through causality-guided attention mechanisms

Innovation

• Defines diagnostic faithfulness metric for model trustworthiness
• Uses causality-guided attention mechanism for anomaly detection
• Implements adversarial consistency learning for improved reliability