Synthesis-in-the-Loop Evaluation of LLMs for RTL Generation: Quality, Reliability, and Failure Modes

2026-03-11

AI Summary

This work addresses the critical gap in existing large language models for RTL generation, which typically evaluate only functional correctness while neglecting synthesizability and hardware implementation quality. We propose the first comprehensive closed-loop evaluation framework for RTL generation, automatically synthesizing outputs from 32 models across 202 Verilog tasks using the Nangate45 process technology. To quantify hardware quality, we introduce the Hardware Quality Index (HQI), which integrates metrics such as area, delay, and synthesis warnings. We further establish a failure taxonomy encompassing 195 distinct real-world synthesis failures, revealing three performance tiers among models—with Gemini-3-Pro leading at 85.1 HQI—and uncover systematic differences in failure patterns between proprietary and open-source models. The evaluation demonstrates high consistency across three standard cell libraries (Spearman ρ > 0.99).

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

RTL generation demands more than software code synthesis: designs must be syntactically valid, synthesizable, functionally correct, and hardware-efficient. Existing evaluations stop at functional correctness, leaving synthesizability and implementation quality unmeasured. We evaluate 32 language models on 202 Verilog tasks from VerilogEval and RTLLM, with five attempts each, scoring via the Hardware Quality Index (HQI), a 0--100 metric integrating post-synthesis area, delay, and warning count relative to expert references under a Nangate45 45\,nm flow. Three performance tiers emerge: 13 frontier models achieve Global HQI above 71, led by Gemini-3-Pro (87.5\% coverage, 85.1 HQI); 11 mid-tier models cluster at 53--68; 8 fall below 53. The capability-to-deployment gap (best-of-five vs.\ single-attempt) spans 3.8--22.1 HQI points, motivating multi-sample strategies. A tool-adjudicated taxonomy of 195 genuine synthesis failures reveals systematic divergence: proprietary models fail late through elaboration errors and synthesis timeout; open-weight models fail early through missing module wrappers and non-synthesizable constructs, consistent with training on simulation-grade rather than synthesis-grade RTL. Rankings hold across three technology libraries at Spearman~$ρ> 0.99$.

Problem

• RTL generation
• synthesizability
• hardware quality
• large language models
• functional correctness

Innovation

• Synthesis-in-the-Loop
• Hardware Quality Index (HQI)
• RTL generation
• LLM evaluation
• synthesizability