HierSVA: A Data Synthesis Pipeline, Dataset, and Benchmark for LLM-Driven Hierarchical Hardware Formal Verification

2026-06-09

AI Summary

This work addresses the lack of systematic data, evaluation benchmarks, and high-quality SystemVerilog assertion (SVA) generation methods for large language models (LLMs) in hierarchical hardware formal verification. It proposes the first LLM-driven framework tailored for hierarchical RTL verification, integrating RTL preprocessing, an LLM-in-the-loop closed-loop verification pipeline, and a fault injection mechanism. The authors construct a large-scale SVA dataset enriched with hierarchical metadata and introduce a six-dimensional quality assessment framework encompassing syntactic correctness, non-emptiness, specification fidelity, and other criteria. Experiments across 342 modules demonstrate a 67.1% compilation rate for generated assertions, an 82.1% non-emptiness proof rate, and a 70.2% fault detection rate. The deep subset achieves a recall of 0.87 (precision: 0.60), and while agent-based prompting improves certain metrics, it exhibits diminishing returns.

Abstract

We present HierSVA, an integrated suite that combines a pipeline, dataset, and benchmark for LLM-driven hierarchical hardware formal verification. HierSVA-SP pairs an RTL preprocessing toolchain with an LLM-in-the-loop formal verification flow to produce reference SystemVerilog Assertions (SVA) on hierarchical RTL. Applying it to BaseJump STL yields HierSVA-DS, a dataset of 342 modules, with hierarchy metadata and depths 0--9, accompanied by a deep subset of 28 module-bug pairs with natural-language specifications and bug variants. HierSVA-B decomposes assertion quality into six metric axes: syntax correctness, assertion proof success rate, vacuity, specification faithfulness, mutation coverage, and formal core coverage. Applying HierSVA-B to twelve recent LLMs reveals three findings. First, the module-level compile rate is 67.1\%; among generated assertions in evaluable runs, 82.1\% prove non-vacuously, but the corresponding assertion sets detect only 70.2\% of eligible injected faults and cover 36.2\% of the formal core. Second, on 211 evaluable model--module entries in the deep subset, assertion sets flag buggy RTL with 0.87 recall, but 40\% of predicted-buggy outcomes are false positives on correct RTL, limiting precision to 0.60. Third, agentic mode improves S1-style provability and strength metrics, but gains plateau and oscillate. Codes and artifacts are available at \href{https://github.com/HierSVAAnon/HierSVACodeAndArtifacts}{https://github.com/HierSVAAnon/HierSVACodeAndArtifacts}. Dataset is available at \href{https://huggingface.co/datasets/AnonymousHierSVA/HierSVA}{https://huggingface.co/datasets/AnonymousHierSVA/HierSVA}.

Problem

• hierarchical hardware verification
• LLM-driven formal verification
• SystemVerilog Assertions
• RTL
• assertion quality

Innovation

• hierarchical formal verification
• LLM-driven hardware verification
• SystemVerilog Assertions (SVA)
• RTL dataset
• assertion quality benchmark