When Does Trajectory-Level Supervision Permit Efficient Offline Reinforcement Learning?

2026-06-16

AI Summary

Offline reinforcement learning typically relies on stepwise rewards, yet real-world datasets often provide only trajectory-level labels, creating a statistical efficiency bottleneck for policy optimization. This work proposes OPAC, an algorithm that combines implicit reward modeling with a pessimistic Actor-Critic framework to enable effective policy learning under trajectory-level supervision alone, and extends it to settings with preference feedback and generalized trajectory objectives. The study establishes the first statistical theory for this setting, providing matching upper and lower bounds on sample complexity, revealing the fundamental challenge posed by the absence of stepwise rewards, and identifying structural conditions under which efficient learning is achievable. Under both standard and preference-based settings, OPAC attains an error bound of Õ(H²√(C_sa(π*)/n)); when the identified structural conditions hold, the generalized OPAC achieves polynomial sample complexity.

Abstract

Offline reinforcement learning is typically analyzed under process-level reward supervision, yet many sequential decision datasets record only trajectory-level outcomes. We develop a statistical theory for offline policy optimization from such outcome-level supervision. We first study the canonical setting where the target remains the expected cumulative reward, but each offline trajectory provides only a scalar label whose conditional mean is the cumulative return. We propose OPAC, a pessimistic actor-critic algorithm that learns a latent reward model and optimizes a policy from trajectory-level labels. We prove a high-probability guarantee of order $\widetilde O(H^2\sqrt{C_{sa}(π^\star)/n})$ and a matching lower bound, characterizing the sharp statistical cost of replacing process-level rewards with one trajectory-level label. We then extend the principle to preference-based feedback, preserving the leading horizon and concentrability dependence up to preference-model constants. Finally, we study generalized outcome-based offline RL, where both the supervision and the objective are trajectory-level quantities induced by a nonlinear aggregation of latent per-step rewards. This problem is not learnable in general: for all-success objectives, any offline learner may require $Ω(2^H)$ trajectories even with deterministic transitions and constant concentrability. We then identify a tractable regime through two structural coefficients, $κ_μ(σ)$ and $χ_μ(σ)$, capturing information loss in outcome aggregation and generalized Bellman updates, under which generalized OPAC achieves polynomial sample complexity. Together, our results delineate when outcome-level supervision enables sample-efficient offline control and when missing process-level rewards create fundamental statistical barriers.

Problem

• offline reinforcement learning
• trajectory-level supervision
• outcome-based feedback
• sample efficiency
• statistical learnability

Innovation

• offline reinforcement learning
• trajectory-level supervision
• pessimistic actor-critic
• sample complexity
• outcome-based feedback