CTISum: A New Benchmark Dataset For Cyber Threat Intelligence Summarization

2024-08-13

arXiv.org

AI Summary

Current network threat intelligence (CTI) summarization suffers from a lack of high-quality benchmark datasets, hindering advances in factual extraction, analytical insight generation, and attack process modeling. To address this gap, we introduce CTISum—the first open-source, human-annotated benchmark dataset specifically designed for CTI summarization. CTISum comprises real-world CTI reports annotated across multiple dimensions, including a novel fine-grained “attack process summarization” subtask. We propose a three-stage collaborative annotation protocol to ensure annotation consistency and semantic fidelity. Leveraging CTISum, we systematically evaluate state-of-the-art extractive (e.g., BERTSum) and generative (e.g., BART, T5) summarization models. Results reveal significant performance limitations of current SOTA methods, confirming the task’s inherent difficulty and establishing a new evaluation standard. This work fills a critical data void in the CTI domain and lays a foundational resource for automated CTI understanding and response.

Technology Category

• Natural Language Processing: Summarization
• Data Mining & Knowledge Management: Data Visualization & Summarization
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Cyber Threat Intelligence (CTI) summarization task requires the system to generate concise and accurate highlights from raw intelligence data, which plays an important role in providing decision-makers with crucial information to quickly detect and respond to cyber threats in the cybersecurity domain. However, efficient techniques for summarizing CTI reports, including facts, analytical insights, attack processes, etc., have largely been unexplored, primarily due to the lack of available dataset. To this end, we present CTISum, a new benchmark for CTI summarization task. Considering the importance of attack process, a novel fine-grained subtask of attack process summarization is proposed to enable defenders to assess risk, identify security gaps, vulnerabilities, and so on. Specifically, we first design a multi-stage annotation pipeline to gather and annotate the CTI data, and then benchmark the CTISum with a collection of extractive and abstractive summarization methods. Experimental results show that current state-of-the-art models exhibit limitations when applied to CTISum, underscoring the fact that automatically producing concise summaries of CTI reports remains an open research challenge.

Problem

• Lack of suitable datasets for CTI report summarization
• Need for attack process summarization in cybersecurity
• Challenges in automatic summarization of CTI reports

Innovation

• Introduces CTISum benchmark dataset for CTI summarization
• Proposes multi-stage annotation pipeline for data collection
• Benchmarks extractive, abstractive, and LLMs-based summarization methods