Detecting Flow Gaps in Data Streams

2025-05-20

AI Summary

This paper addresses the real-time detection of traffic gaps in data streams, introducing the novel paradigm of “value-change monitoring.” Methodologically, we propose GapFilter, a lightweight sketch-based framework that incorporates a similarity absorption algorithm to reduce memory overhead and a dual-layer civilian-suspect classification mechanism to jointly enable coarse-grained pre-filtering and fine-grained refinement. We further instantiate two variants: GapFilter-SO (speed-optimized) and GapFilter-AO (accuracy-optimized). Theoretical analysis establishes rigorous error bounds and memory-accuracy trade-offs: GapFilter-AO achieves benchmark-level accuracy using only 1/32 the memory, while GapFilter-SO attains a 3× throughput improvement. All source code is publicly released.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Search and Optimization: Distributed Search
• Computer Vision: Motion & Tracking

Application Category

• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Data stream monitoring is a crucial task which has a wide range of applications. The majority of existing research in this area can be broadly classified into two types, monitoring value sum and monitoring value cardinality. In this paper, we define a third type, monitoring value variation, which can help us detect flow gaps in data streams. To realize this function, we propose GapFilter, leveraging the idea of Sketch for achieving speed and accuracy. To the best of our knowledge, this is the first work to detect flow gaps in data streams. Two key ideas of our work are the similarity absorption technique and the civilian-suspect mechanism. The similarity absorption technique helps in reducing memory usage and enhancing speed, while the civilian-suspect mechanism further boosts accuracy by organically integrating broad monitoring of overall flows with meticulous monitoring of suspicious flows.We have developed two versions of GapFilter. Speed-Oriented GapFilter (GapFilter-SO) emphasizes speed while maintaining satisfactory accuracy. Accuracy-Oriented GapFilter (GapFilter-AO) prioritizes accuracy while ensuring considerable speed. We provide a theoretical proof demonstrating that GapFilter secures high accuracy with minimal memory usage. Further, extensive experiments were conducted to assess the accuracy and speed of our algorithms. The results reveal that GapFilter-AO requires, on average, 1/32 of the memory to match the accuracy of the Straw-man solution. GapFilter-SO operates at a speed 3 times faster than the Straw-man solution. All associated source code has been open-sourced and is available on GitHub.

Problem

• Detecting flow gaps in data streams
• Monitoring value variation for gap detection
• Achieving speed and accuracy with GapFilter

Innovation

• Uses Sketch for speed and accuracy
• Employs similarity absorption technique
• Implements civilian-suspect mechanism