Evaluating the Effectiveness of LLMs in Aiding Compliance Testing of PKCS#1-v1.5

2026-06-16

AI Summary

Existing approaches struggle to efficiently test semantic compliance of TLV-encoded binary protocols against specifications, and specialized tools often require substantial manual effort. This work proposes a general-purpose testing framework that integrates grammar-level mutation with large language model (LLM)-based code synthesis, and presents the first systematic evaluation of LLMs for cryptographic protocol compliance testing. Applied to PKCS#1 v1.5 signature verification, the method reproduced 10 out of 13 known violation classes—including all 5 signature forgery types—across 48 cryptographic libraries and uncovered one previously unknown vulnerability. Experimental results indicate that LLM hallucination remains a key bottleneck, with semantic fidelity at only 17.5%, though generated test scripts demonstrated 99.8% execution reliability.

Abstract

Testing implementations of binary protocols for specification compliance requires inputs that satisfy both structural and semantic constraints. Purely random generation and primitive mutations are often insufficient for exploring semantically meaningful behaviors in protocols that rely on Type-Length-Value (TLV) encoding, yet domain-specific compliance testing tools require deep protocol expertise and significant manual effort to construct. This work investigates whether grammar-level mutation combined with LLM-based code synthesis can serve as a viable, more generalizable approach to specification compliance testing. We evaluate the approach on PKCS#1 v1.5 signature verification -- a widely deployed TLV-encoded standard with a formally verified testing oracle (Morpheus) -- across 48 cryptographic library implementations. We reproduced 10 of 13 non-trivial specification violation categories previously identified by Morpheus, including all 5 signature forgery categories, and discovered 1 previously unreported discrepancy. We found that LLM hallucination -- occurring in 82.5% of generated scripts -- is the primary factor limiting effectiveness, not the mutation strategies. We identify five distinct hallucination types and show that their distribution varies systematically across mutation categories: structural mutations are implemented with 13.3% fidelity while constraint mutations achieve 30.3% correctness but suffer the highest rate of mutations being fully ignored (8.1%). These findings reveal a striking gap between operational reliability (99.8%) and semantic fidelity (17.5%), providing actionable guidance on when LLM-based code synthesis can be trusted in specification-driven testing pipelines.

Problem

• compliance testing
• PKCS#1 v1.5
• LLM hallucination
• TLV encoding
• specification conformance

Innovation

• LLM-based code synthesis
• grammar-level mutation
• specification compliance testing
• PKCS#1 v1.5
• hallucination analysis