MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmark

2026-01-07

arXiv.org

AI Summary

This work addresses the lack of hardware description language (HDL) support in existing code completion benchmarks, which predominantly focus on general-purpose programming languages and overlook multi-language, repository-scale scenarios. To bridge this gap, we propose MHRC-Bench—the first repository-level code completion benchmark tailored for multi-language HDLs—covering three mainstream HDL coding styles. Built upon concrete syntax trees (CSTs), MHRC-Bench introduces a fine-grained structural and hardware semantic labeling framework that enables multi-language HDL processing, repository-level context modeling, and automated semantic annotation. Comprehensive evaluations across diverse models demonstrate the benchmark’s effectiveness, establishing MHRC-Bench as a standardized platform for developing and evaluating large language models for hardware code.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Hardware-aware ML
• Cognitive Modeling & Cognitive Systems: (Computational) Cognitive Architectures

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models (LLMs) have achieved strong performance on code completion tasks in general-purpose programming languages. However, existing repository-level code completion benchmarks focus almost exclusively on software code and largely overlook hardware description languages. In this work, we present \textbf{MHRC-Bench}, consisting of \textbf{MHRC-Bench-Train} and \textbf{MHRC-Bench-Eval}, the first benchmark designed for multilingual hardware code completion at the repository level. Our benchmark targets completion tasks and covers three major hardware design coding styles. Each completion target is annotated with code-structure-level and hardware-oriented semantic labels derived from concrete syntax tree analysis. We conduct a comprehensive evaluation of models on MHRC-Bench-Eval. Comprehensive evaluation results and analysis demonstrate the effectiveness of MHRC-Bench.

Problem

• hardware description languages
• code completion
• repository-level
• multilingual
• benchmark

Innovation

• hardware description languages
• repository-level code completion
• multilingual benchmark
• code structure annotation
• LLM evaluation