Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis

2025-02-19

AI Summary

Hardware description language (HDL) generation faces key bottlenecks—including scarce training data, high error rates, and inefficient long-sequence modeling—hindering large language model (LLM)-driven hardware design automation. Method: This work systematically establishes high-level synthesis (HLS) as a more LLM-appropriate abstraction than low-level HDL for automated hardware design. We propose an end-to-end LLM-HLS generation framework integrating chain-of-thought prompting and iterative feedback optimization. Contribution/Results: We introduce HLSBench—the first LLM-oriented HLS benchmark—along with a dedicated fine-tuning dataset and automated evaluation pipeline. Using Llama/CodeLlama, we perform instruction tuning integrated with C++/Catapult HLS flow. Experiments demonstrate that our fine-tuned models significantly improve functional correctness and synthesizability of generated HLS code, outperforming general-purpose code models. This work establishes a new paradigm and foundational infrastructure for LLM-powered hardware design automation.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Recent advances in code generation have illuminated the potential of employing large language models (LLMs) for general-purpose programming languages such as Python and C++, opening new opportunities for automating software development and enhancing programmer productivity. The potential of LLMs in software programming has sparked significant interest in exploring automated hardware generation and automation. Although preliminary endeavors have been made to adopt LLMs in generating hardware description languages (HDLs), several challenges persist in this direction. First, the volume of available HDL training data is substantially smaller compared to that for software programming languages. Second, the pre-trained LLMs, mainly tailored for software code, tend to produce HDL designs that are more error-prone. Third, the generation of HDL requires a significantly higher number of tokens compared to software programming, leading to inefficiencies in cost and energy consumption. To tackle these challenges, this paper explores leveraging LLMs to generate High-Level Synthesis (HLS)-based hardware design. Although code generation for domain-specific programming languages is not new in the literature, we aim to provide experimental results, insights, benchmarks, and evaluation infrastructure to investigate the suitability of HLS over low-level HDLs for LLM-assisted hardware design generation. To achieve this, we first finetune pre-trained models for HLS-based hardware generation, using a collected dataset with text prompts and corresponding reference HLS designs. An LLM-assisted framework is then proposed to automate end-to-end hardware code generation, which also investigates the impact of chain-of-thought and feedback loops promoting techniques on HLS-design generation. Limited by the timeframe of this research, we plan to evaluate more advanced reasoning models in the future.

Problem

• LLMs for HLS-based hardware generation
• Challenges in HDL training data volume
• Efficiency in token usage for HDL

Innovation

• LLMs for HLS hardware generation
• Fine-tuning pre-trained models
• End-to-end hardware code automation