Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reports

2026-06-16

AI Summary

This study addresses the limitations of existing evaluation methods for cyber threat intelligence (CTI) report analysis, which rely on oversimplified data and thus overestimate model performance by failing to capture the complexity of multi-label MITRE ATT&CK technique classification in real-world CTI. To bridge this gap, the authors construct the first realistic CTI benchmark dataset comprising 2,076 expert-annotated sentences, accompanied by a rigorous six-stage annotation pipeline. They systematically evaluate seven open-source large language models (ranging from 8B to 236B parameters) under diverse prompting strategies and temperature settings. Results reveal that even the best-performing model achieves only a micro-averaged F1 score of 0.22, highlighting significant limitations of current open-source LLMs in practical CTI scenarios. Model performance shows a strong positive correlation with parameter count, while variations in prompting strategies and temperature yield no substantial gains. The work releases a high-quality dataset and a reproducible evaluation framework to support future research.

Abstract

Classifying Cyber Threat Intelligence (CTI) using MITRE Adversarial Tactics, Techniques, and Common Knowledge (ATT&CK) is essential for proactive defense, but historically required extensive human effort. Pre-Large Language Model (LLM) automation sped up this process, but could not resolve the complex language and multi-step attack patterns found in unstructured CTI reports. LLMs addressed previous limitations by using contextual reasoning to understand unstructured text. However, current evaluations rely on simplified, single-technique sentences that ignore the complexity of real-world CTI reports, which often leads to inflated performance results. Consequently, the baseline performance of open-source LLMs on complex unstructured CTI reports remains unevaluated. To address this gap, we constructed a ground-truth dataset of 2,076 human-annotated sentences (1,281 technique-positive, 795 negative) from 83 complex unstructured CTI reports. These sentences were mapped to 114 unique ATT&CK techniques using a six-phase annotation process, achieving \k{appa} = 0.68 inter-annotator agreement. Using this dataset, we evaluated seven open-source LLMs ranging from 8B to 236B parameters across prompt strategy and temperature configurations. The highest-performing LLM achieved a micro-averaged F1 score of 0.22, establishing the empirical baseline for multi-label ATT&CK classification on complex unstructured CTI. Parameter size showed a statistically significant positive correlation with F1 score. Prompt strategy and temperature produced no statistically significant gains across model configurations. These results indicate that current open-source LLMs are insufficient for production-grade ATT&CK classification. The dataset, benchmark, and findings provide a reproducible foundation for future CTI research.

Problem

• Multi-Label Classification
• ATT&CK
• Cyber Threat Intelligence
• Large Language Models
• Unstructured Text

Innovation

• multi-label classification
• ATT&CK technique
• open-source LLMs
• Cyber Threat Intelligence
• empirical benchmark