Evaluating Large Language Models for Symbolic Security Protocol Analysis
This study presents the first systematic evaluation of the feasibility of using large language models (LLMs) as substitutes for formal verification tools—such as ProVerif and OFMC—in symbolic security protocol analysis. The authors assess GPT and DeepSeek in both chat and reasoning modes on 130 obfuscated AnB/AnBx protocols, comparing their outputs against formal verification results across 388 security properties. In reasoning mode, LLMs achieve a peak F1 score of 95.7% on confidentiality tasks, yet exhibit less than 50% detection accuracy for authentication attacks. These findings highlight the potential of LLMs in confidentiality analysis while underscoring their current limitations in handling authentication properties. Although LLMs cannot yet replace formal methods, they show promise as effective pre-screening tools in protocol analysis workflows.