Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization

2026-07-06

AI Summary

This work addresses the significant performance degradation of reinforcement learning policies under observation delays, a common challenge in real-world settings. In stochastic Markov decision processes, delayed observations inherently diverge from the true current state, introducing bias that impairs policy effectiveness. The paper presents the first theoretical characterization of this bias and introduces a novel delay-aware policy optimization framework. This framework explicitly models the mapping from delayed observations to the current true state using a diffusion model and incorporates an uncertainty-aware mechanism to reweight and correct policy updates. Evaluated on a range of continuous robotic control tasks with stochastic and long observation delays, the proposed method substantially outperforms existing approaches, demonstrating both strong robustness and superior performance.

Abstract

Reinforcement learning in real world environments often suffers from severe performance degradation due to delayed feedback. Existing approaches typically mitigate performance degradation caused by observation delays by constructing augmented states or predicting the true states. However, these methods often overlook the inherent discrepancy between delayed state and true states induced by stochastic MDP. We theoretically prove the existence of such a discrepancy and show that it leads to the degradation of the optimal policy. To address this challenge, we propose Diffusion Guided Uncertainty Aware Delayed Policy Optimization (DUPO). Our method explicitly models the relationship between delayed state message and the current state using a diffusion model, and leverages the resulting discrepancy estimates to weight delayed policies. Extensive experiments on continuous robotic control tasks with multiple stochastic delays demonstrate that DUPO consistently outperforms existing methods and remains effective even under long and random delay scenarios.

Problem

• delayed feedback
• stochastic MDP
• state discrepancy
• reinforcement learning
• policy degradation

Innovation

• diffusion model
• delayed reinforcement learning
• uncertainty-aware policy
• stochastic MDP
• state discrepancy