HWE-Bench: Can Language Models Perform Board-level Schematic Designs?

2026-03-18

AI Summary

This work presents the first systematic evaluation of large language models (LLMs) in board-level circuit schematic design—a task demanding integrated understanding of physical laws and integrated circuit (IC) datasheet knowledge. To this end, we construct a benchmark comprising 300 real-world design tasks and 2,914 IC datasheets, and introduce a dual verification mechanism combining electrical rule checking (static) and SPICE simulation (dynamic). Experimental results show that even the best-performing model achieves only an 8.15% overall pass rate, revealing that while LLMs exhibit preliminary capability in interpreting engineering documentation, they critically lack physical intuition and remain unreliable for complex hardware design. This study establishes the first standardized evaluation framework for AI-driven electronic design automation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) have demonstrated significant potential in various engineering tasks, including software development, digital logic generation, and companion document maintenance. However, their ability to perform board-level circuit design is understudied, as this task requires a synergized understanding of real-world physics and Integrated Circuit (IC) datasheets, the latter comprising detailed specifications for individual components. To address this challenge, we propose \hweb, an evaluation framework that benchmarks the ability of LLMs to perform such designs. It consists of 300 board-level design tasks pulled from open-source and crowdsourcing platforms such as GitHub and OSHWLab, covering 8 application domains, and is complemented with a knowledge base of 2,914 real IC datasheets. For each task, the LLMs are tasked with generating a schematic from scratch, using the provided circuit functional requirements and a set of component datasheets as input. The resulting schematic will be checked against a static electrical rules, and then passed to a circuit simulator to verify its dynamic behavior. Our evaluation show that although current models achieve initial engineering usability and documentation understanding, they lack physical intuition, as the top-performing model achieved an overall pass rate of 8.15\%. We envision that advancements on \hweb\ will pave the way for the development of practical Electronic Design Automation (EDA) agents, revolutionizing the field of board-level design.

Problem

• board-level schematic design
• Large Language Models
• IC datasheets
• Electronic Design Automation
• circuit design

Innovation

• HWE-Bench
• board-level schematic design
• large language models
• IC datasheets
• circuit simulation