Fitted Occupancy-Ratio Evaluation without Bellman Completeness

2026-07-06

AI Summary

This work addresses the policy evaluation bias in offline reinforcement learning caused by distributional shift by proposing Fitted Occupancy Ratio Estimation (FORE). The method formulates a fixed-point equation for the occupancy ratio via an adjoint Bellman recursion and iteratively optimizes a density ratio objective under KL divergence using single-step transition data. Its key innovation lies in establishing the realizability of the occupancy ratio as a sufficient condition, thereby circumventing reliance on strong assumptions such as Bellman completeness. Theoretical analysis shows that FORE converges to the true occupancy ratio under KL divergence and achieves a finite-sample regret bound solely under this realizability condition. Moreover, the approach accommodates multiple value estimation schemes and exhibits both double robustness and strong empirical performance.

Abstract

Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax methods typically estimate these ratios by enforcing occupancy-balance moments over a critic class. We propose fitted occupancy-ratio evaluation (FORE), a fitted fixed-point method that characterizes the discounted occupancy ratio through an adjoint Bellman recursion. At each iteration, FORE solves a single-level density-ratio objective on one-step-transition data, thereby projecting the adjoint Bellman image onto a log-ratio class in Kullback--Leibler (KL) divergence. Unlike analyses of fitted Q-evaluation, which typically require value-function realizability together with Bellman completeness or projected-operator stability, our central approximation condition is just realizability of the discounted occupancy ratio itself. Under this condition, the population KL-projected recursion contracts in relative entropy toward the true ratio by virtue of the adjoint Bellman operator being a KL-contraction. For the empirical recursion, we establish finite-sample regret bounds that yield convergence in KL up to log-ratio approximation error and a statistical error governed by the complexity of the ratio hypothesis class. The fitted ratio supports direct value estimation by reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation that combines the fitted ratio with a fitted Q-function. Together, these results identify discounted occupancy-ratio realizability as a sufficient condition for offline policy evaluation without any completeness assumptions.

Problem

• offline reinforcement learning
• off-policy evaluation
• occupancy ratio
• Bellman completeness
• distribution shift

Innovation

• offline reinforcement learning
• occupancy ratio
• adjoint Bellman operator
• fitted evaluation
• Bellman completeness