Model-free reinforcement learning with noisy actions for automated experimental control in optics

2024-05-24

arXiv.org

AI Summary

This work addresses the challenging control problem of highly noisy and strongly nonlinear optical systems—specifically, laser coupling into single-mode fibers. We propose a model-free reinforcement learning (RL) approach that eliminates reliance on system identification or physical modeling. For the first time, we directly deploy Soft Actor-Critic (SAC) augmented with Truncated Quantile Critics (TQC) in a real-world optical experimental setup, enabling end-to-end online training and closed-loop control. The method learns an optimal policy solely through environmental interaction, without prior dynamical modeling or parameter estimation. Experimental results demonstrate stable achievement of 90% coupling efficiency—matching the performance of expert human operators—while exhibiting strong robustness to noise and nonlinearity. This validates the feasibility, robustness, and engineering practicality of model-free RL for complex, real-time optical control tasks.

Technology Category

• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Setting up and controlling optical systems is often a challenging and tedious task. The high number of degrees of freedom to control mirrors, lenses, or phases of light makes automatic control challenging, especially when the complexity of the system cannot be adequately modeled due to noise or non-linearities. Here, we show that reinforcement learning (RL) can overcome these challenges when coupling laser light into an optical fiber, using a model-free RL approach that trains directly on the experiment without pre-training. By utilizing the sample-efficient algorithms Soft Actor-Critic (SAC) or Truncated Quantile Critics (TQC), our agent learns to couple with 90% efficiency, comparable to the human expert. We demonstrate that direct training on an experiment can replace extensive system modeling. Our result exemplifies RL's potential to tackle problems in optics, paving the way for more complex applications where full noise modeling is not feasible.

Problem

• Automated control of optical systems
• Model-free reinforcement learning application
• Overcoming noise and non-linearities in optics

Innovation

• Model-free reinforcement learning
• Direct experimental training
• Soft Actor-Critic algorithm