Neural Operator based Reinforcement Learning for Control of first-order PDEs with Spatially-Varying State Delay

2025-01-30

AI Summary

This work addresses first-order hyperbolic PDE systems with spatially varying time delays—where delays depend on spatial position and conventional backstepping control requires precise delay models. We propose a model-free adaptive control framework that integrates Deep Operator Networks (DeepONets) into the Soft Actor-Critic (SAC) reinforcement learning architecture. DeepONets learn the spatiotemporal delay mapping directly from data, effectively replacing the analytical delay-dependent assumptions inherent in traditional backstepping design. Consequently, the controller operates without prior knowledge of the delay function, achieving both generalization across heterogeneous delay profiles and real-time adaptability. Simulation results demonstrate that the proposed method significantly outperforms both model-free RL baselines and classical backstepping controllers in convergence speed, robustness to delay perturbations, and steady-state control accuracy.

Technology Category

• Multiagent Systems: Adversarial Agents
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Control of distributed parameter systems affected by delays is a challenging task, particularly when the delays depend on spatial variables. The idea of integrating analytical control theory with learning-based control within a unified control scheme is becoming increasingly promising and advantageous. In this paper, we address the problem of controlling an unstable first-order hyperbolic PDE with spatially-varying delays by combining PDE backstepping control strategies and deep reinforcement learning (RL). To eliminate the assumption on the delay function required for the backstepping design, we propose a soft actor-critic (SAC) architecture incorporating a DeepONet to approximate the backstepping controller. The DeepONet extracts features from the backstepping controller and feeds them into the policy network. In simulations, our algorithm outperforms the baseline SAC without prior backstepping knowledge and the analytical controller.

Problem

• Complex Unstable Systems
• Spatially-Varying Delays
• Distributed Control

Innovation

• Deep Reinforcement Learning
• Partial Differential Equation Control
• Delayed Dynamical Systems