Adaptive Reinforcement Learning for Unobservable Random Delays

2025-06-17

AI Summary

Real-world dynamic systems often exhibit unobservable, stochastic, time-varying interaction delays—violating the standard Markov Decision Process (MDP) assumption of instantaneous observation and action execution. Method: This paper proposes a reinforcement learning framework that operates without prior knowledge of delay bounds and enables online, adaptive modeling and response to such delays. Its core components are: (1) a generic interaction layer explicitly modeling delay propagation; (2) a model-based Actor-Critic architecture integrating delay-aware action matrix generation and implicit delay distribution estimation; and (3) a dynamic policy adjustment mechanism for delay-robust decision-making. Results: Evaluated on diverse biomimetic locomotion benchmarks, the method significantly outperforms state-of-the-art approaches—achieving a 37% improvement in robustness and a 2.1× increase in tolerance to dropped actions. It is the first approach to enable prior-free, online adaptive control under unknown, time-varying delays.

Technology Category

• Multiagent Systems: Adversarial Agents
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

In standard Reinforcement Learning (RL) settings, the interaction between the agent and the environment is typically modeled as a Markov Decision Process (MDP), which assumes that the agent observes the system state instantaneously, selects an action without delay, and executes it immediately. In real-world dynamic environments, such as cyber-physical systems, this assumption often breaks down due to delays in the interaction between the agent and the system. These delays can vary stochastically over time and are typically unobservable, meaning they are unknown when deciding on an action. Existing methods deal with this uncertainty conservatively by assuming a known fixed upper bound on the delay, even if the delay is often much lower. In this work, we introduce the interaction layer, a general framework that enables agents to adaptively and seamlessly handle unobservable and time-varying delays. Specifically, the agent generates a matrix of possible future actions to handle both unpredictable delays and lost action packets sent over networks. Building on this framework, we develop a model-based algorithm, Actor-Critic with Delay Adaptation (ACDA), which dynamically adjusts to delay patterns. Our method significantly outperforms state-of-the-art approaches across a wide range of locomotion benchmark environments.

Problem

• Handling unobservable random delays in RL environments
• Adapting to time-varying delays without fixed upper bounds
• Managing unpredictable delays and lost action packets

Innovation

• Adaptive Reinforcement Learning for unobservable delays
• Interaction layer framework for time-varying delays
• Actor-Critic with Delay Adaptation algorithm