🤖 AI Summary
This study addresses the lack of effective multimodal evaluation benchmarks for large language models (LLMs) in hardware RTL design, which hinders accurate assessment of the functional correctness and structural plausibility of generated circuit diagrams. To bridge this gap, the authors introduce the first multimodal benchmark comprising 99 diverse RTL modules and propose a multi-stage evaluation framework that integrates rule-based scoring, self-evaluation, cross-model peer review, blind assessment, and human verification. Experimental results reveal that while current LLMs can produce visually interpretable circuit diagrams, their functional correctness remains limited. Moreover, automated evaluations exhibit almost no agreement with human judgments, highlighting the unreliability of using LLMs as evaluators in structured engineering tasks and underscoring the need for domain-specific assessment methodologies.
📝 Abstract
Recently, Large Language models (LLMs) find application in several fields. This extends to hardware definition and synthesis. However, most works at the intersection of LLMs and hardware generation focus on text-based tasks, creating a gap for multi-modal LLMs for RTL design. In this work, we introduce MultModLM, a benchmark for evaluating LLMs on the task of generating hardware schematics from RTL (Register Transfer Level) descriptions. The dataset consists of 99 diverse RTL modules spanning arithmetic, control, and state-based designs. To address the challenges of non-unique schematic representations, we propose a multi-stage evaluation framework combining rubric-based scoring, self-evaluation, cross-model assessment, blind evaluation, and human validation to enable exhaustive evaluation.
Through experiments on state-of-the-art LLMs, we observe that while models can generate visually interpretable schematics, their functional correctness remains constrained. Furthermore, we find that LLM-based evaluators exhibit near-zero agreement with human raters, revealing, as a key finding, that LLM-as-a-judge paradigms are unreliable in structurally precise domains. These findings suggest that reliable evaluation of multi-modal hardware outputs remains an open challenge, motivating the need for more robust and domain-aware evaluation methodologies, as well as tools for structural evaluation, so as to enable formal equivalence checkers.