PICBench: Benchmarking LLMs for Photonic Integrated Circuits Design

2025-02-05

AI Summary

Photonic integrated circuit (PIC) design relies heavily on repetitive, error-prone manual coding with limited automation. To address this, we introduce PICBench—the first LLM evaluation benchmark specifically targeting PIC design automation, centered on netlist generation across diverse tasks ranging from elementary photonic components to complex circuits. PICBench establishes a novel dual-dimension automated evaluation framework assessing both syntactic correctness and functional correctness; it is the first to integrate open-source photonic simulators (SAX/NumPy) for end-to-end functional verification, enabling reproducible and scalable benchmarking. Comprehensive evaluation of state-of-the-art LLMs—including GPT and Llama series—reveals critical limitations in physical consistency and functional accuracy. PICBench thus provides a foundational benchmark and actionable insights for advancing intelligent photonic EDA tools.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search
• Computer Vision: Computational Photography, Image & Video Synthesis

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

While large language models (LLMs) have shown remarkable potential in automating various tasks in digital chip design, the field of Photonic Integrated Circuits (PICs)-a promising solution to advanced chip designs-remains relatively unexplored in this context. The design of PICs is time-consuming and prone to errors due to the extensive and repetitive nature of code involved in photonic chip design. In this paper, we introduce PICBench, the first benchmarking and evaluation framework specifically designed to automate PIC design generation using LLMs, where the generated output takes the form of a netlist. Our benchmark consists of dozens of meticulously crafted PIC design problems, spanning from fundamental device designs to more complex circuit-level designs. It automatically evaluates both the syntax and functionality of generated PIC designs by comparing simulation outputs with expert-written solutions, leveraging an open-source simulator. We evaluate a range of existing LLMs, while also conducting comparative tests on various prompt engineering techniques to enhance LLM performance in automated PIC design. The results reveal the challenges and potential of LLMs in the PIC design domain, offering insights into the key areas that require further research and development to optimize automation in this field. Our benchmark and evaluation code is available at https://github.com/PICDA/PICBench.

Problem

• Automating Photonic Integrated Circuits design
• Benchmarking LLMs for PIC design generation
• Evaluating syntax and functionality of PIC designs

Innovation

• LLMs automate PIC design
• Benchmarking framework for PICs
• Open-source simulation evaluation