When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning

2026-04-23

AI Summary

This work addresses the challenge of safely adapting frozen policies to new objectives during deployment in offline reinforcement learning. The authors propose a deployment-time adaptation method based on a Product-of-Experts (PoE) framework combined with a goal-conditioned prior, which enables anchored guidance of the frozen policy through precision-weighted fusion. Theoretical analysis establishes an equivalence between PoE composition and KL-regularized adaptation under a frozen policy constraint, yielding a closed-form relationship that informs the design of a KL budget selector to approximate the optimal operating point. Experiments on the D4RL benchmark demonstrate the method's efficacy, revealing a HELP/FROZEN/HURT distribution across 4/5/3 tasks, respectively. These results indicate that PoE and KL-based mechanisms can jointly serve as safe anchoring strategies, while also exposing a theoretical upper bound on policy adaptability.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Multiagent Systems: Adversarial Agents

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Offline reinforcement learning (RL) can learn effective policies from fixed datasets, but deployment objectives may change after training, and in many applications the trained actor cannot be retrained because of data, cost, or governance constraints. We study deployment-time adaptation for frozen offline actors using Product-of-Experts (PoE) composition with a goal-conditioned prior. Our main practical finding is graceful degradation rather than universal performance gain: under degraded or random priors, precision-weighted composition remains anchored to the frozen actor, while additive and prior-only adaptation collapse, and a KL-budget selector often recovers a near-oracle operating point. We also make explicit a closed-form identity in the frozen-actor setting: for diagonal-Gaussian actors and priors, PoE with coefficient alpha yields the same deterministic policy as KL-regularized adaptation with beta = alpha / (1 - alpha), with posterior covariances differing only by a global scalar factor. Empirically, across four D4RL environments (3,900 MuJoCo episodes), we observe a 4/5/3 HELP/FROZEN/HURT split. Extending the analysis to six harder cells and two AntMaze diagnostics reveals an actor-competence ceiling: medium-expert remains HURT in all 9 cells at every tested alpha, while AntMaze with a behavior-cloned frozen actor yields zero success for all composition rules. Overall, PoE and KL-regularized adaptation are best viewed as a single actor-anchored safety mechanism for deployment-time steering.

Problem

• offline reinforcement learning
• deployment-time adaptation
• frozen policy
• policy steering
• goal-conditioned prior

Innovation

• Product-of-Experts
• KL-regularized adaptation
• closed-form equivalence
• frozen policy steering
• offline reinforcement learning