TuRTLe: A Unified Evaluation of LLMs for RTL Generation

2025-03-31

AI Summary

Existing RTL code generation benchmarks inadequately evaluate LLMs across syntax correctness, functional accuracy, synthesizability, and PPA (performance, power, area) optimization. This paper introduces the first unified, automated evaluation framework tailored for hardware design—incorporating synthesis feasibility and PPA constraints into LLM assessment for the first time. The framework integrates Verilator-based functional simulation, Yosys-based logic synthesis, PPA analysis, and line-level code completion evaluation. Systematic evaluation reveals distinct model capabilities: inference-optimized models (e.g., DeepSeek R1) achieve superior overall metrics but incur high latency; base models excel at module-level code completion; and instruction-tuned models outperform others in specification-to-RTL translation. These findings establish a new empirical benchmark and methodological paradigm for evaluating LLMs in EDA applications.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Evaluation and Analysis

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The rapid advancements in LLMs have driven the adoption of generative AI in various domains, including Electronic Design Automation (EDA). Unlike traditional software development, EDA presents unique challenges, as generated RTL code must not only be syntactically correct and functionally accurate but also synthesizable by hardware generators while meeting performance, power, and area constraints. These additional requirements introduce complexities that existing code-generation benchmarks often fail to capture, limiting their effectiveness in evaluating LLMs for RTL generation. To address this gap, we propose TuRTLe, a unified evaluation framework designed to systematically assess LLMs across key RTL generation tasks. TuRTLe integrates multiple existing benchmarks and automates the evaluation process, enabling a comprehensive assessment of LLM performance in syntax correctness, functional correctness, synthesis, PPA optimization, and exact line completion. Using this framework, we benchmark a diverse set of open LLMs and analyze their strengths and weaknesses in EDA-specific tasks. Our results show that reasoning-based models, such as DeepSeek R1, consistently outperform others across multiple evaluation criteria, but at the cost of increased computational overhead and inference latency. Additionally, base models are better suited in module completion tasks, while instruct-tuned models perform better in specification-to-RTL tasks.

Problem

• Evaluating LLMs for RTL code generation in EDA
• Assessing syntax, function, and synthesis constraints
• Comparing model performance in EDA-specific tasks

Innovation

• Unified evaluation framework for RTL generation
• Automates syntax, function, synthesis, PPA assessment
• Benchmarks diverse LLMs for EDA-specific tasks