VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning

2025-04-16

AI Summary

In offline reinforcement learning, model-based approaches often rely on unreliable heuristic uncertainty estimates due to dynamics model errors, leading to overly conservative policies. To address this, we propose a novel model calibration paradigm that explicitly enforces intrinsic consistency between the learned dynamics model and the value function. Specifically, we introduce a self-supervised value inconsistency regularizer—computed as the discrepancy between dual-network value estimates—and minimize a consistency loss during model training. This formulation directly constrains model predictions to align with value function expectations, improving model accuracy without requiring explicit uncertainty modeling. The method eliminates the need for auxiliary uncertainty estimation modules and exhibits strong generalization across domains. Evaluated on all benchmark tasks in D4RL and NeoRL, our approach achieves state-of-the-art performance, significantly improving both model prediction accuracy and policy return.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Sequential Decision Making
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Offline reinforcement learning (RL) learns effective policies from pre-collected datasets, offering a practical solution for applications where online interactions are risky or costly. Model-based approaches are particularly advantageous for offline RL, owing to their data efficiency and generalizability. However, due to inherent model errors, model-based methods often artificially introduce conservatism guided by heuristic uncertainty estimation, which can be unreliable. In this paper, we introduce VIPO, a novel model-based offline RL algorithm that incorporates self-supervised feedback from value estimation to enhance model training. Specifically, the model is learned by additionally minimizing the inconsistency between the value learned directly from the offline data and the one estimated from the model. We perform comprehensive evaluations from multiple perspectives to show that VIPO can learn a highly accurate model efficiently and consistently outperform existing methods. It offers a general framework that can be readily integrated into existing model-based offline RL algorithms to systematically enhance model accuracy. As a result, VIPO achieves state-of-the-art performance on almost all tasks in both D4RL and NeoRL benchmarks.

Problem

• Addresses unreliable heuristic uncertainty in model-based offline RL
• Reduces inconsistency between data-derived and model-estimated value functions
• Enhances model accuracy for state-of-the-art offline RL performance

Innovation

• Self-supervised feedback enhances model training
• Minimizes value inconsistency for accurate models
• Integrates into existing model-based offline RL