AttackonCTF: Defending Hardware Security Competition Benchmarks in the Age of LLMs

2026-06-14

AI Summary

This work reveals that large language models (LLMs) in hardware security benchmarks often exploit syntactic differences rather than genuine security reasoning, leading to inflated vulnerability detection rates and undermining benchmark validity. To address this, the authors propose the first semantics-preserving obfuscation framework tailored for LLMs, which applies human-readable, functionally equivalent code transformations combined with controlled diff-based noise injection. This approach effectively suppresses LLMs’ reliance on non-semantic cues while preserving code readability and functionality. Evaluated on the HackTheSilicon benchmark, the method reduces LLM detection accuracy by 50% with only 10% obfuscation and by up to 78.6% under full obfuscation, substantially restoring the reliability and fairness of hardware security evaluations.

Abstract

Hardware security competitions such as HackTheSilicon serve as benchmarking platforms for evaluating vulnerability detection methods and for training humans and AI. However, our study reveals that LLMs threaten their validity. Instead of genuine security reasoning, detectors exploit a diff-style syntactic comparison, achieving an 83% detection rate, undermining fair evaluation. To mitigate this, we propose the first LLM-oriented, semantics-preserving obfuscation framework for these benchmarks. Unlike IP-protection approaches, it applies human-readable transformations and controlled diff-noise while preserving functionality. On HackTheSilicon, the framework reduces LLM-based detection accuracy by 50% with only 10% obfuscation and by 78.6% under complete obfuscation, restoring benchmark reliability.

Problem

• Hardware security
• LLM vulnerability detection
• Benchmark validity
• Syntactic comparison
• Security competition

Innovation

• LLM-aware obfuscation
• semantics-preserving transformation
• hardware security benchmark
• vulnerability detection
• diff-style syntactic comparison