CrypFormBench: Benchmarking Formal Analysis Capability of Large Language Models for Cryptographic Schemes

2026-06-24

AI Summary

This study addresses the lack of standardized evaluation methods for assessing large language models (LLMs) in formal cryptographic protocol analysis, a gap that hinders their application in symbolic and computational security verification. To bridge this gap, we introduce CrypFormBench, the first comprehensive benchmark dedicated to formal cryptographic analysis, comprising 700 instances, 677 cryptographic protocols, seven formal specification languages, and 160 security properties. We systematically evaluate LLMs across five core tasks: interpretation, generation, completion, translation, and correction. Our methodology integrates multi-dimensional task design with formal tools such as Scyther, Tamarin, and CryptoVerif, enhanced by few-shot prompting, Pass@K sampling, and lightweight fine-tuning to improve output usability. Evaluation of nine prominent LLMs reveals relatively stronger performance in interpretation and completion tasks, yet overall capabilities remain limited—peaking at 48.7/100 for Claude-3.5—highlighting critical avenues for future improvement.

Abstract

Manual formal analysis of cryptographic schemes is labor-intensive and requires substantial expertise. While model-checking tools (e.g., Scyther and Tamarin) and computational-security tools (e.g., CryptoVerif and EasyCrypt) improve the automation of security proofs, they still rely on experts to abstract schemes and write tool-specific formal descriptions. Large language models (LLMs) are a promising alternative, but their effectiveness in this domain remains unexplored due to the absence of standardized evaluation methodologies. To fill this gap, we introduce CrypFormBench (C.F.B for short), a comprehensive benchmark jointly covering symbolic and computational security to evaluate five core LLM capabilities: interpretation, generation, completion, transformation, and correction. It comprises 700 instances spanning 677 schemes, 7 mainstream formal verifier languages, and 160 security properties. The evaluation of 9 state-of-the-art LLMs reveals that most of them perform well on interpretation and completion, given their code-awareness advantages, but struggle with generation, transformation, and correction. Overall, their performance remains limited, with Claude-3.5 achieving the highest score at 48.7 out of 100. We further provide practical guidance, e.g., few-shot prompting, Pass@K sampling, and lightweight fine-tuning, to mitigate the executability bottleneck and improve tool-usable outputs. Taken together, our benchmark and analyses offer a grounded view of current progress and concrete directions toward reliable LLM-assisted formal cryptographic analysis.

Problem

• large language models
• formal analysis
• cryptographic schemes
• benchmarking
• security proofs

Innovation

• formal verification
• large language models
• cryptographic protocols
• benchmarking
• symbolic and computational security