OPTIAGENT: A Physics-Driven Agentic Framework for Automated Optical Design

2026-02-27

AI Summary

This work addresses the long-standing reliance on expert intuition in optical design and the inability of conventional methods to automate the process effectively. While existing large language models possess general optical knowledge, they fail to generate physically realizable lens systems. To bridge this gap, we propose a physics-driven agent framework that integrates optical priors with the reasoning capabilities of large language models, enabling automatic generation of functional lenses without task-specific training. Our key innovations include an optical lexicographic reward mechanism—enforcing structural validity, physical feasibility, and ray-tracing accuracy—a hybrid training objective, and a DrGRPO-based policy optimization strategy. We further introduce the OptiDesignQA dataset and an end-to-end optimization pipeline. Experiments demonstrate that our approach significantly outperforms both traditional automated design algorithms and current large-model methods, successfully producing high-fidelity, physically consistent lens systems across multiple benchmarks.

Technology Category

• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV
• Multiagent Systems: Mechanism Design

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Optical design is the process of configuring optical elements to precisely manipulate light for high-fidelity imaging. It is inherently a highly non-convex optimization problem that relies heavily on human heuristic expertise and domain-specific knowledge. While Large Language Models (LLMs) possess extensive optical knowledge, their capabilities in leveraging the knowledge in designing lens system remain significantly constrained. This work represents the first attempt to employ LLMs in the field of optical design. We bridge the expertise gap by enabling users without formal optical training to successfully develop functional lens systems. Concretely, we curate a comprehensive dataset, named OptiDesignQA, which encompasses both classical lens systems sourced from standard optical textbooks and novel configurations generated by automated design algorithms for training and evaluation. Furthermore, we inject domain-specific optical expertise into the LLM through a hybrid objective of full-system synthesis and lens completion. To align the model with optical principles, we employ Group Relative Policy Optimization Done Right (DrGRPO) guided by Optical Lexicographic Reward for physics-driven policy alignment. This reward system incorporates structural format rewards, physical feasibility rewards, light-manipulation accuracy, and LLM-based heuristics. Finally, our model integrates with specialized optical optimization routines for end-to-end fine-tuning and precision refinement. We benchmark our proposed method against both traditional optimization-based automated design algorithms and LLM counterparts, and experimental results show the superiority of our method.

Problem

• optical design
• non-convex optimization
• large language models
• domain expertise
• lens system

Innovation

• Physics-Driven LLM
• Optical Design Automation
• OptiDesignQA Dataset
• Lexicographic Reward Optimization
• Agentic Framework