Beyond Penalization: Diffusion-based Out-of-Distribution Detection and Selective Regularization in Offline Reinforcement Learning

2026-05-05

AI Summary

This work addresses the challenge in offline reinforcement learning of accurately identifying out-of-distribution (OOD) actions while avoiding the over-suppression of beneficial exploration caused by uniform penalty strategies. The authors propose DOSER, a novel framework that introduces diffusion models for OOD detection: two diffusion models are employed to separately model the behavior policy and state transition dynamics, and single-step denoising reconstruction errors are leveraged to detect OOD actions. During policy optimization, predicted state transitions are used to distinguish between beneficial and harmful OOD actions, enabling selective regularization. Theoretical analysis establishes the algorithm’s γ-contraction property and performance guarantees. Experiments demonstrate that DOSER significantly outperforms existing methods across multiple offline RL benchmarks, exhibiting particularly strong effectiveness and robustness on suboptimal datasets.

Abstract

Offline reinforcement learning (RL) faces a critical challenge of overestimating the value of out-of-distribution (OOD) actions. Existing methods mitigate this issue by penalizing unseen samples, yet they fail to accurately identify OOD actions and may suppress beneficial exploration beyond the behavioral support. Although several methods have been proposed to differentiate OOD samples with distinct properties, they typically rely on restrictive assumptions about the data distribution and remain limited in discrimination ability. To address this problem, we propose DOSER (Diffusion-based OOD Detection and Selective Regularization), a novel framework that goes beyond uniform penalization. DOSER trains two diffusion models to capture the behavior policy and state distribution, using single-step denoising reconstruction error as a reliable OOD indicator. During policy optimization, it further distinguishes between beneficial and detrimental OOD actions by evaluating predicted transitions, selectively suppressing risky actions while encouraging exploration of high-potential ones. Theoretically, we prove that DOSER is a $γ$-contraction and therefore admits a unique fixed point with bounded value estimates. We further provide an asymptotic performance guarantee relative to the optimal policy under model approximation and OOD detection errors. Across extensive offline RL benchmarks, DOSER consistently attains superior performance to prior methods, especially on suboptimal datasets.

Problem

• offline reinforcement learning
• out-of-distribution detection
• value overestimation
• behavioral support
• selective regularization

Innovation

• diffusion models
• out-of-distribution detection
• selective regularization
• offline reinforcement learning
• denoising reconstruction error