Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experience

2026-06-25

AI Summary

Direct reinforcement learning in real-world environments is often prohibitively costly, while unconstrained simulation-based training suffers from domain gaps that hinder transferability, and conventional regularization methods can impede policy improvement. To address these challenges, this work proposes the SCORE framework, which enforces support-set constraints to restrict the policy to actions executable by a base policy, enabling safe and efficient optimization in simulation without modifying the base policy or relying on dense rewards. By integrating generative pre-training, support-set constraints, and flow-based guidance, SCORE establishes a novel “real-to-sim-to-real” paradigm. Evaluated on eight dexterous manipulation tasks, the method significantly improves success rates from 37.8% to 89.9%—outperforming the best baseline by 59.5%—and reduces task completion steps by 36.8%, demonstrating its effectiveness and strong transferability.

Abstract

Robots trained on real world data tend to be imprecise, slow, and brittle to perturbations. Improving these policies with reinforcement learning (RL) is an appealing alternative, but this process often requires expensive training in the real world. Performing policy improvement in simulation instead provides a far cheaper alternative, but unconstrained RL in simulation can exploit contact and dynamics mismatches, resulting in unsafe behaviors that do not transfer to hardware. Common forms of regularization can furthermore limit improvement by overconstraining to an imperfect behavior prior. In this work, we propose Support-Constrained Off-Domain REinforcement (SCORE), a real-to-sim-to-real framework that constrains RL in simulation to the support of a generative policy pretrained on real data. We instantiate this constraint through flow steering, restricting SCORE to actions the base policy can already produce, which ensures transferable behaviors while maximizing policy improvement. Improving a policy with SCORE requires minimal effort: it learns from sparse rewards, avoids distillation, and leaves the base policy untouched. Across eight real-world dexterous multi-fingered robotic manipulation tasks, SCORE improves average success rate from 37.8% to 89.9%, compared to 59.5% for the best baseline, and reaches success in 36.8% fewer steps than the base policy. Ultimately, through extensive experiments and ablations, we show that simulation can substantially improve real-world manipulation policies when policy optimization is appropriately constrained, introducing a new paradigm for real-to-sim-to-real policy improvement. Videos and code are available at https://weirdlabuw.github.io/score/.

Problem

• real-to-sim-to-real
• policy improvement
• reinforcement learning
• simulation-to-reality transfer
• support constraint

Innovation

• Support-Constrained RL
• real-to-sim-to-real
• flow steering
• policy improvement
• off-domain reinforcement learning