VHDLSuite: Unified Pipeline for LLM VHDL Generation with Data Synthesis and Evaluation

2026-06-11

AI Summary

This work addresses the limited generalization capability of large language models (LLMs) across hardware description languages, particularly due to the absence of a systematic evaluation framework for VHDL. We propose the first unified framework for LLM-based VHDL generation and evaluation, introducing an automated, verifiable Verilog-to-VHDL benchmark conversion pipeline. The resulting VHDLBench dataset comprises over 200 VHDL modules, each accompanied by complete testbenches. Integrating automated data synthesis, the VUnit/GHDL verification toolchain, and multi-model comparative analysis, our framework enables the first comprehensive assessment of LLM-generated VHDL code in terms of compilability, executability, and functional correctness. This study reveals critical challenges posed by VHDL-specific semantics and structural constructs, laying the groundwork for multilingual hardware design automation.

Abstract

Large Language Models (LLM) have shown impressive capabilities in Register Transfer Level (RTL) code generation, particularly for Verilog. However, evaluating their performance with other Hardware Description Languages (HDL), especially VHDL, remains limited although its distinct language characteristics, such as stricter semantic rules, introduce evaluation considerations that differ from Verilog. This lack of coverage restricts fully understanding of how well current models generalize across hardware design languages with differing structures and semantics. To address this gap, we introduce VHDLSuite, a benchmark-centered infrastructure for scalable VHDL generation evaluation, integrating automated benchmark synthesis, executable validation, and multi-model diagnostic analysis. First, we propose a data pipeline that automatically converts Verilog designs and their accompanying testbenches into executable VHDL benchmark instances, followed by VUnit/GHDL-based validation to ensure each released task is compilable, runnable, and consistently checkable in the VHDL environment. Second, we introduce VHDLBench, a benchmark with over 200 VHDL problems with complete and validated testbenches across a wide range of complexity levels. Third, we extensively evaluate cutting-edge LLMs and uncover key challenges specific on LLM-aided VHDL generation. Our findings provide important insights and support future work in multi-language hardware design automation.Our data pipeline, benchmark, and evaluation framework will be open-sourced.

Problem

• VHDL
• Large Language Models
• Hardware Description Languages
• RTL code generation
• model generalization

Innovation

• VHDL generation
• LLM evaluation
• automated benchmark synthesis
• hardware description languages
• VHDLBench