OIPR: Evaluation for Time-series Anomaly Detection Inspired by Operator Interest

2025-03-03

AI Summary

This paper addresses the imbalance between point-based and event-based metrics in time-series anomaly detection (TAD) evaluation: point-level metrics overestimate performance on prolonged anomalies, while event-level metrics are sensitive to fragmented false positives. To resolve this, we propose the Operator-Interest Process-based Recall (OIPR) framework—a novel human-centered evaluation paradigm that explicitly incorporates operator cognitive preferences. OIPR models operator attention toward anomaly onset/offset, duration, and severity to construct an interest curve; the area under this curve jointly quantifies both point-level precision and event-level plausibility. Experiments on a custom challenging scenario and five real-world datasets demonstrate that OIPR significantly enhances evaluation robustness and practicality—particularly under extreme or complex conditions—while effectively mitigating overfitting to long anomalies and fragmentation-induced false detections.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Evaluation and Analysis
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Security and Privacy: Large-scale security measurements

Abstract

With the growing adoption of time-series anomaly detection (TAD) technology, numerous studies have employed deep learning-based detectors for analyzing time-series data in the fields of Internet services, industrial systems, and sensors. The selection and optimization of anomaly detectors strongly rely on the availability of an effective performance evaluation method for TAD. Since anomalies in time-series data often manifest as a sequence of points, conventional metrics that solely consider the detection of individual point are inadequate. Existing evaluation methods for TAD typically employ point-based or event-based metrics to capture the temporal context. However, point-based metrics tend to overestimate detectors that excel only in detecting long anomalies, while event-based metrics are susceptible to being misled by fragmented detection results. To address these limitations, we propose OIPR, a novel set of TAD evaluation metrics. It models the process of operators receiving detector alarms and handling faults, utilizing area under the operator interest curve to evaluate the performance of TAD algorithms. Furthermore, we build a special scenario dataset to compare the characteristics of different evaluation methods. Through experiments conducted on the special scenario dataset and five real-world datasets, we demonstrate the remarkable performance of OIPR in extreme and complex scenarios. It achieves a balance between point and event perspectives, overcoming their primary limitations and offering applicability to broader situations.

Problem

• Develops OIPR metrics for time-series anomaly detection evaluation.
• Addresses limitations of point-based and event-based evaluation methods.
• Enhances performance assessment in extreme and complex scenarios.

Innovation

• OIPR models operator alarm handling process
• Uses area under operator interest curve
• Balances point and event detection perspectives