Orthogonalized Estimation of Difference of $Q$-functions

2024-06-12

AI Summary

In offline reinforcement learning, accurately estimating the action-value function difference $Q^pi(s,1) - Q^pi(s,0)$ from static historical data—enabling optimal multi-action policy selection—is a fundamental challenge. This paper introduces the first dynamic extension of the R-learner from causal inference to RL, proposing an orthogonalized estimation framework for offline Q-difference estimation. The method is robust to slowly converging auxiliary models (e.g., black-box Q-function and behavior policy estimators), ensures consistent policy optimization under mild marginal conditions, and achieves faster convergence rates. Theoretically, it establishes estimation consistency without requiring strong parametric assumptions on auxiliary models. Empirically, the approach directly enables optimal multi-action policy selection and significantly improves offline decision-making performance across benchmark tasks.

Technology Category

• Machine Learning: Online Learning & Bandits
• Intelligent Robots: Learning & Optimization for ROB
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Social networks and social learning
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Offline reinforcement learning is important in many settings with available observational data but the inability to deploy new policies online due to safety, cost, and other concerns. Many recent advances in causal inference and machine learning target estimation of causal contrast functions such as CATE, which is sufficient for optimizing decisions and can adapt to potentially smoother structure. We develop a dynamic generalization of the R-learner (Nie and Wager 2021, Lewis and Syrgkanis 2021) for estimating and optimizing the difference of $Q^pi$-functions, $Q^pi(s,1)-Q^pi(s,0)$ (which can be used to optimize multiple-valued actions). We leverage orthogonal estimation to improve convergence rates in the presence of slower nuisance estimation rates and prove consistency of policy optimization under a margin condition. The method can leverage black-box nuisance estimators of the $Q$-function and behavior policy to target estimation of a more structured $Q$-function contrast.

Problem

• Estimates difference of Q-functions for offline RL
• Optimizes policies using orthogonal learning
• Improves convergence with nuisance estimation

Innovation

• Orthogonal estimation for faster convergence
• Dynamic R-learner generalization for Q-difference
• Black-box nuisance estimators for structured contrast