OOD-RL-Bench: A Benchmark Framework for Out-of-Distribution Detection in Reinforcement Learning

2026-07-14

AI Summary

This work addresses the challenge that reinforcement learning (RL) agents struggle to effectively detect anomalies in observations or trajectory dynamics under sensor failures or distribution shifts, a gap exacerbated by existing out-of-distribution (OOD) detection benchmarks that overlook the action-dependent temporal structure inherent in RL. To bridge this gap, the authors introduce OOD-RL-Bench, the first OOD detection benchmark tailored for RL, which injects diverse action-conditioned temporal anomalies—such as observation perturbations, mechanism switches, and observation delays—into trajectories to enable plug-and-play evaluation. Using the LunarLander-v3 environment with a DQN policy, the study conducts systematic assessments via multidimensional metrics including AUROC, AUPRC, false positive rate, and detection delay. Results reveal substantial differences in detectability across anomaly types. The benchmark framework, models, and results are publicly released.

Abstract

Reliable reinforcement learning (RL) agents must maintain operational integrity amidst sensor malfunctions, dynamic disturbances, and slow environmental shifts. The detection of out-of-distribution conditions is pivotal to determining when an agent's observations, transitions, or trajectory dynamics deviate from the assumptions underpinning its policy training. Current out-of-distribution (OOD) detection benchmarks typically evaluate image classifiers or static low-dimensional datasets, failing to account for the complex, action-dependent temporal structure inherent in RL trajectories. To address this gap, we present OOD-RL-Bench, a comprehensive and extensible framework designed to evaluate OOD detectors against categories of anomalies injected into RL trajectories. Detectors and anomaly injectors are integrated through shared interfaces and configuration, which allows new scoring methods and perturbation families to be evaluated without modification of the core benchmark loop. We evaluate the utility of the framework using a Deep Q-Network policy within the LunarLander-v3 environment. We assess the performance of each detector across a suite of anomaly types using matched-time AUROC, matched-time AUPRC, matched-time false-positive rate, detection delay, and segmented-onset metrics. Our analysis reveals significant performance variance across anomaly types: observation perturbations and regime switches are identified with high accuracy by several methods, while observation delay and action-conditioned dynamics remain difficult even when post-onset anomaly scores are compared against clean scores from the same timesteps. We make the framework, trained policy checkpoint, and complete results publicly available as a reproducible artefact.

Problem

• out-of-distribution detection
• reinforcement learning
• anomaly detection
• temporal dynamics
• benchmark

Innovation

• out-of-distribution detection
• reinforcement learning
• benchmark framework
• temporal anomaly injection
• action-conditioned dynamics