LLM-Aided Testbench Generation and Bug Detection for Finite-State Machines

2024-06-24

arXiv.org

AI Summary

To address low efficiency, insufficient coverage, and poor RTL bug detection in FSM-based chip functional verification, this paper proposes an EDA-feedback-driven, closed-loop LLM testbench generation method. Initial testbenches are generated using GPT-3.5 or GPT-4; then, real-time signal-level feedback—including code and state coverage metrics and error diagnostics—from commercial EDA tools (e.g., Synopsys VCS) is integrated into the prompt engineering process, enabling iterative refinement. This work pioneers deep integration of EDA tool feedback into the LLM generation pipeline, supporting coverage-guided automated test generation and concurrent RTL-level bug detection. Evaluated on multiple industrial-grade FSM designs, the method improves code and state coverage by 20–35% over baseline approaches and successfully identifies timing and control-logic bugs missed by manual verification. The approach significantly enhances both verification efficiency and reliability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This work investigates the potential of tailoring Large Language Models (LLMs), specifically GPT3.5 and GPT4, for the domain of chip testing. A key aspect of chip design is functional testing, which relies on testbenches to evaluate the functionality and coverage of Register-Transfer Level (RTL) designs. We aim to enhance testbench generation by incorporating feedback from commercial-grade Electronic Design Automation (EDA) tools into LLMs. Through iterative feedback from these tools, we refine the testbenches to achieve improved test coverage. Our case studies present promising results, demonstrating that this approach can effectively enhance test coverage. By integrating EDA tool feedback, the generated testbenches become more accurate in identifying potential issues in the RTL design. Furthermore, we extended our study to use this enhanced test coverage framework for detecting bugs in the RTL implementations

Problem

• Enhancing testbench generation using LLMs for chip testing
• Improving test coverage with EDA tool feedback integration
• Detecting bugs in RTL designs via enhanced testbenches

Innovation

• LLM-aided testbench generation for chip testing
• EDA tool feedback enhances test coverage
• Improved bug detection in RTL designs