Threat Modelling using Domain-Adapted Language Models: Empirical Evaluation and Insights

2026-05-11

AI Summary

Current large language models exhibit inconsistent performance in structured threat modeling, and the efficacy of domain adaptation remains unclear. This study presents the first systematic evaluation of eight language models—including both domain-adapted and general-purpose variants across a range of sizes—on their ability to classify threats according to the STRIDE framework in 5G security scenarios. Spanning 52 experimental configurations, the work comprehensively investigates the impact of model scale, domain adaptation, decoding strategies, and prompting techniques. The findings reveal that domain adaptation does not consistently enhance performance, decoding strategy significantly affects output validity, and gains from increased model size are limited. Based on these empirical results, the paper proposes STRIDE-specific prompt design guidelines to support reliable automated threat modeling.

Abstract

Large Language Models(LLMs) are increasingly explored for cybersecurity applications such as vulnerability detection. In the domain of threat modelling, prior work has primarily evaluated a number of general-purpose Large Language Models under limited prompting settings. In this study, we extend the research area of structured threat modelling by systematically evaluating domain-adapted language models of different sizes to their general counterparts. We use both LLMs and Small Language Models(SLMs) that were domain adapted to telecommunications and cybersecuirty. For the structured threat modelling, we selected the widely used STRIDE approach and the application area is 5G security. We present a comprehensive empirical evaluation using 52 different configurations (on 8 different language models) to analyze the impact of 1) domain adaptation, 2) model scale, 3) decoding strategies (greedy vs. stochastic sampling), and 4) prompting technique on STRIDE threat classification. Our results show that domain-adapted models do not consistently outperform their general-purpose counterparts, and decoding strategies significantly affect model behavior and output validity. They also show that while larger models generally achieve higher performance, these gains are neither consistent nor sufficient for reliable threat modelling. These findings highlight fundamental limitations of current LLMs for structured threat modelling tasks and suggest that improvements require more than additional training data or model scaling, motivating the need for incorporating more task-specific reasoning and stronger grounding in security concepts. We present insights on invalid outputs encountered and present suggestions for prompting tailored specifically for STRIDE threat modelling.

Problem

• Threat Modelling
• Large Language Models
• Domain Adaptation
• STRIDE
• 5G Security

Innovation

• domain-adapted language models
• structured threat modelling
• STRIDE
• 5G security
• empirical evaluation