Distributed Incast Detection in Data Center Networks

2025-11-04

AI Summary

Incast traffic in datacenter networks severely degrades performance, yet existing queue-length-based detection methods suffer from high latency and excessive false positives. This paper proposes a switch-level distributed incast detection mechanism that innovatively leverages the statistical properties of inter-arrival times of new flows to formulate a probabilistic hypothesis test. By integrating micro-burst traffic modeling with dynamic optimal threshold optimization, the method enables real-time, accurate incast identification starting from the first packet. It requires no per-flow state maintenance and incurs minimal computational overhead, making it inherently suitable for distributed architectures. Experimental evaluation demonstrates that, compared to state-of-the-art approaches—including MA-ECN, BurstRadar, and Pulser—our method reduces average detection latency by 62%, improves accuracy by 19.3%, and suppresses the false positive rate to below 0.8%.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Planning, Routing, and Scheduling: Scheduling under Uncertainty

Application Category

• Web Mining and Content Analysis: Content-based information diffusion
• Security and Privacy: Large-scale security measurements
• Responsible Web: Measurement, analysis, and circumvention of Web censorship

Abstract

Incast traffic in data centers can lead to severe performance degradation, such as packet loss and increased latency. Effectively addressing incast requires prompt and accurate detection. Existing solutions, including MA-ECN, BurstRadar and Pulser, typically rely on fixed thresholds of switch port egress queue lengths or their gradients to identify microburst caused by incast flows. However, these queue length related methods often suffer from delayed detection and high error rates. In this study, we propose a distributed incast detection method for data center networks at the switch-level, leveraging a probabilistic hypothesis test with an optimal detection threshold. By analyzing the arrival intervals of new flows, our algorithm can immediately determine if a flow is part of an incast traffic from its initial packet. The experimental results demonstrate that our method offers significant improvements over existing approaches in both detection speed and inference accuracy.

Problem

• Detecting incast traffic causing performance degradation
• Overcoming delayed detection and high error rates
• Identifying incast flows instantly from initial packets

Innovation

• Distributed incast detection using probabilistic hypothesis test
• Analyzing flow arrival intervals for immediate incast identification
• Optimal threshold selection enhances detection speed and accuracy