A streaming algorithm and hardware accelerator for top-K flow detection in network traffic

2025-11-20

AI Summary

To address the challenge of efficiently and accurately identifying Top-K flows under skewed traffic distributions in high-speed networks, this paper proposes a streaming algorithm that integrates a dynamically updated TowerSketch with a parallel priority-queue array, implemented as a high-throughput hardware accelerator on an AMD Virtex-U280 FPGA. Our contributions are threefold: (1) an enhanced TowerSketch adapted to dynamic flow spectra, significantly improving estimation accuracy under skew; (2) a parallelizable priority-queue array design that alleviates on-chip memory bottlenecks; and (3) line-rate processing capability ≥200 Gbps—achieving one packet per cycle. Experimental evaluation on real-world traces demonstrates Top-K identification accuracy exceeding 0.94 and average relative error in frequency estimation below 1.96%.

Technology Category

• Data Mining & Knowledge Management: Data Stream Mining
• Machine Learning: Hardware-aware ML
• Search and Optimization: Distributed Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Identifying the largest K flows in network traffic is an important task for applications such as flow scheduling and anomaly detection, which aim to improve network efficiency and security. However, accurately estimating flow frequencies is challenging due to the large number of flows and increasing network speeds. Hardware accelerators are often used in this endeavor due to their high computational power, but their limited amount of on-chip memory constrains their performance. Various sketch-based algorithms have been proposed to estimate properties of traffic such as frequency, with lower memory usage and theoretical bounds, but they often under perform with the skewed distribution of network traffic. In this work, we propose an algorithm for top-K identification using a modified TowerSketch and a priority queue array. Tested on real traffic traces, we identify the top-K flows, with K up to 32,768, with a precision of more than 0.94, and estimate their frequency with an average relative error under 1.96%. We designed and implemented an accelerator for this algorithm on an AMD VirtexU280 UltraScale+ FPGA, which processes one packet per cycle at392 MHz, reaching a minimum line rate of more than 200 Gbps.

Problem

• Accurately identifying top-K network flows with limited memory
• Handling skewed traffic distribution in frequency estimation algorithms
• Achieving high-speed packet processing with hardware accelerator constraints

Innovation

• Modified TowerSketch algorithm for top-K flow detection
• Priority queue array for efficient flow frequency tracking
• FPGA hardware accelerator achieving 200 Gbps processing speed