Problem
Research questions and friction points this paper is trying to address.
Detect novelties in RL world models
Protect agents from sudden changes
Improve novelty detection in RL environments
Novelty Detection in Reinforcement Learning with World Models
🤖 AI Summary
To address the abrupt performance degradation of agents in world-model-based reinforcement learning under sudden environmental shifts—such as abrupt changes in visual attributes or transition dynamics—this paper proposes a lightweight, interpretable boundary-based novelty detection mechanism. The method builds upon a VAE-RSSM world model and integrates multi-scale reconstruction error analysis, confidence-interval-based threshold estimation, and transition-dynamics consistency checking. Crucially, it directly models state prediction mismatch from the world model as an anomaly score, enabling real-time, retraining-free detection at the transition-distribution level. Evaluated on a novel benchmark with controlled environmental mutations, our approach achieves a 27% higher detection accuracy and a 41% lower false positive rate compared to state-of-the-art OOD detectors (e.g., ODIN, Mahalanobis) and RL-specific novelty detection methods (e.g., RND, IDS), significantly improving deployment robustness.
📝 Abstract
Reinforcement learning (RL) using world models has found significant recent successes. However, when a sudden change to world mechanics or properties occurs then agent performance and reliability can dramatically decline. We refer to the sudden change in visual properties or state transitions as novelties. Implementing novelty detection within generated world model frameworks is a crucial task for protecting the agent when deployed. In this paper, we propose straightforward bounding approaches to incorporate novelty detection into world model RL agents, by utilizing the misalignment of the world model's hallucinated states and the true observed states as an anomaly score. We provide effective approaches to detecting novelties in a distribution of transitions learned by an agent in a world model. Finally, we show the advantage of our work in a novel environment compared to traditional machine learning novelty detection methods as well as currently accepted RL focused novelty detection algorithms.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Incorporates novelty detection in world models
Uses misalignment as anomaly detection metric
Compares with traditional RL novelty methods
🔎 Similar Papers
No similar papers found.
