🤖 AI Summary
Existing offline reinforcement learning (Offline RL) benchmarks severely underestimate critical challenges in real-world deployment: extremely conservative data distributions, significant action delays, uncontrollable environmental disturbances, and globally defined safety constraints that are difficult to model and verify. To address this gap, we propose NeoRL-2—the first systematic, extended Offline RL benchmark explicitly designed to capture these four industrial-grade challenges. Built upon high-fidelity physics simulation and controllable perturbation generation, NeoRL-2 comprises seven realistic tasks, each accompanied by an evaluation-capable simulator supporting behavioral policy comparison, formal safety verification, and generalization assessment. Empirical evaluation reveals that mainstream Offline RL algorithms consistently fail to outperform the original behavior policies, exposing fundamental limitations in robustness and safety compliance. NeoRL-2 establishes a standardized, reproducible evaluation platform and benchmark baseline to advance the reliable, industrial-scale deployment of Offline RL.
📝 Abstract
Offline reinforcement learning (RL) aims to learn from historical data without requiring (costly) access to the environment. To facilitate offline RL research, we previously introduced NeoRL, which highlighted that datasets from real-world tasks are often conservative and limited. With years of experience applying offline RL to various domains, we have identified additional real-world challenges. These include extremely conservative data distributions produced by deployed control systems, delayed action effects caused by high-latency transitions, external factors arising from the uncontrollable variance of transitions, and global safety constraints that are difficult to evaluate during the decision-making process. These challenges are underrepresented in previous benchmarks but frequently occur in real-world tasks. To address this, we constructed the extended Near Real-World Offline RL Benchmark (NeoRL-2), which consists of 7 datasets from 7 simulated tasks along with their corresponding evaluation simulators. Benchmarking results from state-of-the-art offline RL approaches demonstrate that current methods often struggle to outperform the data-collection behavior policy, highlighting the need for more effective methods. We hope NeoRL-2 will accelerate the development of reinforcement learning algorithms for real-world applications. The benchmark project page is available at https://github.com/polixir/NeoRL2.