Dynamic Cluster Analysis to Detect and Track Novelty in Network Telescopes

2024-05-17

2024 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW)

AI Summary

Detecting evolving collaborative host behaviors in network telescopes is challenging due to high noise levels, dynamic behavioral shifts, and the coexistence of legacy and emerging attack patterns. Method: This paper proposes a three-stage dynamic clustering framework: (1) robust traffic representation learning via self-supervised embedding; (2) daily-granularity dynamic K-means clustering; and (3) cross-temporal cluster alignment and evolutionary tracking to re-identify historical patterns and automatically discover novel attack clusters. Contribution/Results: To our knowledge, this is the first approach integrating self-supervised representation learning, dynamic clustering, and temporal cluster evolution modeling. Evaluated on 20 days of real-world telescope data, it identifies 50–70 semantically coherent clusters per day, consistently recovers 60%–70% of known activities, and precisely detects 10–20 previously unseen attack clusters—significantly improving threat detection timeliness and analytical efficiency.

Technology Category

• Machine Learning: Clustering
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Search and Optimization: Evolutionary Computation

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Security and Privacy: Data transparency and provenance

Abstract

In the context of cybersecurity, tracking the activi-ties of coordinated hosts over time is a daunting task because both participants and their behaviours evolve at a fast pace. We address this scenario by solving a dynamic novelty dis-covery problem with the aim of both re-identifying patterns seen in the past and highlighting new patterns. We focus on traffic collected by Network Telescopes, a primary and noisy source for cybersecurity analysis. We propose a 3-stage pipeline: (i) we learn compact representations (embeddings) of hosts through their traffic in a self-supervised fashion; (ii) via clustering, we distinguish groups of hosts performing similar activities; (iii) we track the cluster temporal evolution to highlight novel patterns. We apply our methodology to 20 days of telescope traffic during which we observe more than 8 thousand active hosts. Our results show that we efficiently identify 50–70 well-shaped clusters per day, 60–70 % of which we associate with already analysed cases, while we pinpoint 10–20 previously unseen clusters per day. These correspond to activity changes and new incidents, of which we document some. In short, our novelty discovery methodology enormously simplifies the manual analysis the security analysts have to conduct to gain insights to interpret novel coordinated activities.

Problem

• Detect novelty in network traffic
• Track evolving host activities
• Simplify cybersecurity pattern analysis

Innovation

• Self-supervised host embeddings
• Dynamic clustering for novelty
• Temporal evolution tracking