Merit Network Telescope: Processing and Initial Insights from Nearly 20 Years of Darknet Traffic for Cybersecurity Research

2025-10-28

AI Summary

Understanding the long-term evolution of global cyber threats remains challenging due to the scarcity of large-scale, longitudinal network telescope data. Method: This study leverages 19 years (2005–2024) of darknet traffic collected by Merit Network—the largest and longest-operating network telescope in the U.S.—and proposes a two-stage analytical framework: (i) a metadata-based sub-pipeline for resource-efficient coarse-grained filtering, followed by (ii) a packet-header parsing sub-pipeline enabling fine-grained attack classification. Contribution/Results: The framework achieves scalable processing of petabyte-scale historical traffic. It uncovers previously undocumented periodic surges in scanning activity, distributed denial-of-service (DDoS) attacks, and internet outage-correlated events. Empirically, the study identifies multiple global-scale internet scans and several large-scale DDoS incidents. These findings provide critical longitudinal observational evidence for characterizing the temporal dynamics and spatial patterns of the global threat landscape.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Time-Series/Data Streams
• Search and Optimization: Distributed Search

Application Category

• Responsible Web: Ethical and legal aspects of web-scale data analysis, uses and collection practices
• Web Mining and Content Analysis: Web traffic and log analysis
• Security and Privacy: Large-scale security measurements

Abstract

This paper presents an initial longitudinal analysis of unsolicited Internet traffic collected between 2005 and 2025 by one of the largest and most persistent network telescopes in the United States, operated by Merit Network. The dataset provides a unique view into global threat activity as observed through scanning and backscatter traffic, key indicators of large-scale probing behavior, data outages, and ongoing denial-of-service (DoS) campaigns. To process this extensive archive, coarse-to-fine methodology is adopted in which general insights are first extracted through a resource-efficient metadata sub-pipeline, followed by a more detailed packet header sub-pipeline for finer-grained analysis. The methodology establishes two sub-pipelines to enable scalable processing of nearly two decades of telescope data and supports multi-level exploration of traffic dynamics. Initial insights highlight long-term trends and recurring traffic spikes, some attributable to Internet-wide scanning events and others likely linked to DoS activities.We present general observations spanning 2006-2024, with a focused analysis of traffic characteristics during 2024.

Problem

• Analyzing 20 years of darknet traffic for cybersecurity insights
• Processing longitudinal network telescope data with scalable methodology
• Identifying global threat patterns through scanning and DoS traffic

Innovation

• Coarse-to-fine methodology for scalable data processing
• Two sub-pipelines for multi-level traffic exploration
• Resource-efficient metadata and packet header analysis