🤖 AI Summary
This work addresses the insufficient reliability of large language models in low-level operations on SPICE netlists—a limitation often masked by high-level design reasoning. To rigorously evaluate structural fidelity at the netlist level, the authors introduce NetlistBench, the first benchmark dedicated to netlist structural reliability, comprising 24 task categories and 2,342 test cases. It employs a structure-aware, deterministic verifier to assess model performance on parameter identification, connectivity editing, hierarchical manipulation, and equivalence checking. The study innovatively decouples netlist-level reliability from high-level design tasks and introduces long-span composite editing challenges alongside multi-granularity evaluation strategies. Experiments reveal near-perfect accuracy (96%–100%) on simple edits, but substantial performance drops in device insertion (41%–83%) and equivalence judgment (49%–90%). While reasoning augmentation improves weaker models, maintaining structural consistency in long-span edits remains a critical bottleneck.
📝 Abstract
Large Language Models (LLMs) are increasingly used in circuit design workflows, yet their reliability on simulator-facing SPICE netlist recognition and manipulation remains poorly understood and is rarely separated from high-level design reasoning. Although netlists are textual, they encode structured circuit objects through topology and parameters. We present \textbf{NetlistBench}, a structure-verified benchmark for SPICE netlist recognition and manipulation. NetlistBench contains 2,342 cases across 24 task families, covering parameter and connectivity recognition and edits, hierarchical operations, equivalence judgment, and long-horizon compound editing. Model outputs are evaluated by a deterministic structure-aware oracle. Across six non-thinking LLMs, performance varies substantially with operation-level structural complexity. Simple local edits reach $96\%$--$100\%$ accuracy, while device addition drops to $41\%$--$83\%$ and equivalence judgment to $49\%$--$90\%$. Enabling reasoning substantially improves weaker models but does not eliminate structure-preservation failures, with performance still degrading sharply as the edit horizon increases. NetlistBench identifies netlist reliability as a distinct bottleneck for trustworthy LLM-based circuit design automation.