VeriInteresting: An Empirical Study of Model Prompt Interactions in Verilog Code Generation

2026-02-04

AI Summary

This study systematically investigates the interplay between language model characteristics and prompt design in Verilog code generation. Through factorial experiments, it evaluates the performance of three model categories—general-purpose, reasoning-enhanced, and domain-specialized—across diverse prompting strategies, including structured output formatting, chain-of-thought reasoning, in-context learning, and genetic-Pareto optimized prompts. The work presents the first empirical mapping of model–prompt interactions for Verilog generation, revealing distinct response patterns across two benchmarks. It delineates the boundaries of general prompt effectiveness and identifies specific model–prompt dependencies, offering actionable and generalizable prompt engineering guidelines to support efficient hardware-software co-design.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Hardware-aware ML
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Rapid advances in language models (LMs) have created new opportunities for automated code generation while complicating trade-offs between model characteristics and prompt design choices. In this work, we provide an empirical map of recent trends in LMs for Verilog code generation, focusing on interactions among model reasoning, specialization, and prompt engineering strategies. We evaluate a diverse set of small and large LMs, including general-purpose, reasoning, and domain-specific variants. Our experiments use a controlled factorial design spanning benchmark prompts, structured outputs, prompt rewriting, chain-of-thought reasoning, in-context learning, and evolutionary prompt optimization via Genetic-Pareto. Across two Verilog benchmarks, we identify patterns in how model classes respond to structured prompts and optimization, and we document which trends generalize across LMs and benchmarks versus those that are specific to particular model-prompt combinations.

Problem

• Verilog code generation
• language models
• prompt engineering
• model-prompt interaction
• empirical study

Innovation

• Verilog code generation
• prompt engineering
• language model evaluation
• structured prompting
• Genetic-Pareto optimization