Preserve Support, Not Correspondence: Dynamic Routing for Offline Reinforcement Learning

2026-04-24

AI Summary

This work addresses the challenge in single-step offline reinforcement learning where policies struggle to simultaneously improve Q-values and remain supported by the dataset actions. To this end, the authors propose DROL, a method that samples multiple candidate actions in a latent space and employs a top-1 dynamic routing mechanism to dynamically assign each dataset action to its nearest candidate. Only the winning candidate is updated, effectively integrating behavior cloning with critic guidance. By leveraging conditional latent action generation, bounded latent prior sampling, and dynamic routing, DROL overcomes the limitations of conventional point-to-point matching, enabling adaptive shifts in the ownership of support regions during training. Experimental results demonstrate that DROL significantly outperforms existing methods on OGBench and achieves competitive performance on D4RL tasks such as AntMaze and Adroit.

Technology Category

• Machine Learning: Reinforcement Learning
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

One-step offline RL actors are attractive because they avoid backpropagating through long iterative samplers and keep inference cheap, but they still have to improve under a critic without drifting away from actions that the dataset can support. In recent one-step extraction pipelines, a strong iterative teacher provides one target action for each latent draw, and the same student output is asked to do both jobs: move toward higher Q and stay near that paired endpoint. If those two directions disagree, the loss resolves them as a compromise on that same sample, even when a nearby better action remains locally supported by the data. We propose DROL, a latent-conditioned one-step actor trained with top-1 dynamic routing. For each state, the actor samples $K$ candidate actions from a bounded latent prior, assigns each dataset action to its nearest candidate, and updates only that winner with Behavior Cloning and critic guidance. Because the routing is recomputed from the current candidate geometry, ownership of a supported region can shift across candidates over the course of learning. This gives a one-step actor room to make local improvements that pointwise extraction struggles to capture, while retaining single-pass inference at test time. On OGBench and D4RL, DROL is competitive with the one-step FQL baseline, improving many OGBench task groups while remaining strong on both AntMaze and Adroit. Project page: https://muzhancun.github.io/preprints/DROL.

Problem

• offline reinforcement learning
• one-step actor
• action support
• policy improvement
• distributional constraint

Innovation

• dynamic routing
• offline reinforcement learning
• one-step actor
• latent-conditioned policy
• behavior cloning