Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning

2026-06-15

AI Summary

Preference-based reinforcement learning often faces scalability bottlenecks due to its reliance on large amounts of human-annotated feedback. This work proposes the VOTP framework, which for the first time integrates vision foundation models with optimal transport theory to generate high-quality pseudo-reward labels for offline reinforcement learning in a semi-supervised manner, using only a small number of human preference annotations. By efficiently leveraging unlabeled trajectory data, the method demonstrates strong robustness under visual distractions and in real-world robotic tasks, significantly outperforming existing approaches across multiple locomotion and manipulation benchmarks. Notably, VOTP successfully learns high-fidelity reward functions from limited human feedback.

Abstract

Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering. Preference-based RL (PbRL) offers a promising alternative by learning reward functions from human feedback, but its scalability is hindered by high labeling costs. Inspired by advances in Video Foundation Models (ViFMs), we present Video-based Optimal Transport Preference (VOTP), a semi-supervised framework that learns effective reward functions from only a handful of labels. By leveraging optimal transport to align visual trajectories within the rich representation space of ViFMs, VOTP effectively generates high-fidelity pseudo-labels for large amounts of unlabeled data, substantially reducing human supervision. Extensive experiments across locomotion and manipulation benchmarks demonstrate the superiority of VOTP, which outperforms state-of-the-art offline PbRL methods under limited feedback budgets. We also showcase the robustness of VOTP in the presence of visual distractors and validate its utility on real robotic tasks, where it learns meaningful rewards with minimal human input.

Problem

• Preference-based Reinforcement Learning
• Offline Reinforcement Learning
• Reward Learning
• Human Feedback
• Label Efficiency

Innovation

• Video Foundation Models
• Optimal Transport
• Preference-Based Reinforcement Learning
• Semi-supervised Learning
• Offline RL