Automatic Reward Shaping from Confounded Offline Data

2025-05-16

AI Summary

This work addresses offline reinforcement learning under unobserved confounding, where input mismatch between behavior and target policies induces biased policy evaluation. We propose the first deep RL algorithm that integrates worst-case environmental robustness into the offline DQN framework. Our method unifies causal counterfactual robust optimization with minimax policy search, yielding a provably safe reward shaping technique that requires neither observability of confounders nor domain-specific priors. Evaluated on 12 confounded Atari benchmarks, our approach consistently outperforms standard DQN—achieving 37%–62% gains in policy performance under severe input mismatch. The framework establishes a new paradigm for safe and reliable offline policy learning in high-dimensional, nonstationary environments.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

A key task in Artificial Intelligence is learning effective policies for controlling agents in unknown environments to optimize performance measures. Off-policy learning methods, like Q-learning, allow learners to make optimal decisions based on past experiences. This paper studies off-policy learning from biased data in complex and high-dimensional domains where emph{unobserved confounding} cannot be ruled out a priori. Building on the well-celebrated Deep Q-Network (DQN), we propose a novel deep reinforcement learning algorithm robust to confounding biases in observed data. Specifically, our algorithm attempts to find a safe policy for the worst-case environment compatible with the observations. We apply our method to twelve confounded Atari games, and find that it consistently dominates the standard DQN in all games where the observed input to the behavioral and target policies mismatch and unobserved confounders exist.

Problem

• Learning policies from biased offline data with unobserved confounders
• Proposing a robust deep RL algorithm for confounded environments
• Improving performance over DQN in confounded Atari game scenarios

Innovation

• Robust deep reinforcement learning algorithm
• Handles unobserved confounding in data
• Safe policy for worst-case environment