How Far Should We Need to Go : Evaluate Provenance-based Intrusion Detection Systems in Industrial Scenarios

2026-03-24

AI Summary

This study addresses the lack of systematic evaluation of provenance-based intrusion detection systems (PIDS) in industrial settings. For the first time, we construct a multi-source heterogeneous dataset from real-world industrial environments and simulate advanced persistent threat (APT) attacks to comprehensively benchmark five state-of-the-art PIDS approaches. Our evaluation reveals three critical characteristics and challenges unique to industrial scenarios: poor cross-platform portability, low true-positive detection rates for real attacks, and high false-positive rates. To mitigate the latter, we propose a novel method that substantially reduces false alerts by approximately two-thirds, thereby decreasing the manual verification workload proportionally. Furthermore, our findings offer several actionable directions for optimizing PIDS deployment and performance in industrial contexts.

Technology Category

• Application Domains: Internet of Things, Sensor Networks & Smart Cities
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Multiagent Systems: Other Foundations of Multi Agent Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Security and Privacy: Large-scale security measurements

Abstract

Provenance-based Intrusion Detection Systems (PIDSes) have been widely used to detect Advanced Persistent Threats (APTs). Although many studies achieve high performance in the evaluations of their original papers, their performance in industrial scenarios remains unclear. To fill this gap, we conduct the first systematic evaluation and analysis of PIDSes in industrial scenarios. We first analyze the differences between the data from DARPA datasets and that collected in industrial scenarios, identifying three main new characteristics in industry: heterogeneous multi-source inputs, more powerful attackers, and increasing benign activity complexity. We then build several datasets to evaluate five state-of-the-art PIDSes. The evaluation results reveal challenges for existing PIDSes, including poor portability across different hosts and platforms, low detection performance against real-world attacks, and high false positive rates with ever-changing benign activities. Based on the evaluation results and our industrial practices, we provide several insights to solve or explain the above problems. For example, we propose a method to mitigate the high false positives, which reduces manual effort by 2/3. Finally, we propose several research suggestions to improve PIDSes.

Problem

• Provenance-based Intrusion Detection Systems
• Industrial Scenarios
• Advanced Persistent Threats
• False Positive Rate
• System Evaluation

Innovation

• Provenance-based Intrusion Detection
• Industrial Evaluation
• Advanced Persistent Threats
• False Positive Reduction
• Heterogeneous Data