CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning

2026-07-02

AI Summary

This work addresses the challenges of handcrafting reward functions in reinforcement learning and the inefficiency and instability often encountered in preference-based methods. The authors propose a self-supervised preference alignment framework that decomposes the reward into an explicit formal reward (FR) and an implicit residual reward (RR). The FR is iteratively refined by a Formal Reward Module (FRM) guided by a vision-language model to encode task-specific knowledge, while the RR is learned at the video level by a Residual Reward Module (RRM) to capture semantic aspects not explicitly represented by the FR. Evaluated across ten simulated robotic manipulation tasks and five real-world scenarios, the proposed method significantly outperforms existing approaches, achieving notable improvements in both policy performance and sample efficiency.

Abstract

Reward design remains a central challenge in reinforcement learning (RL). Hand-crafted rewards are often difficult to specify and may lead to suboptimal policies, while learned rewards from preferences can suffer from inefficiency and unstable training. Inspired by the dual nature of human learning explored in cognitive science, we decompose rewards into two complementary components: Formal Rewards (FR), explicitly designed based on task knowledge, and Residual Rewards (RR), learned from observations to capture implicit and nuanced preferences. Based on this decomposition, we propose CoRe, a hybrid framework that integrates FR and RR with vision-language models (VLMs) feedback to achieve preference-aligned policies without human involvement. Our contributions are twofold: (1) We propose a Formal Reward Module (FRM) that leverages VLMs to iteratively design and optimize FR based on task knowledge and preference feedback, enabling the continual improvement of policy during training; (2) We introduce a Residual Reward Module (RRM) that learns RR from video-level preference by employing VLMs to generate preference labels and capturing nuanced rewards that complement FR, ensuring alignment with human intent. Through the synergy of FRM and RRM, CoRe enables the automatic construction of reliable rewards that are efficient and preference-aligned. Extensive experiments demonstrate that CoRe outperforms existing approaches in terms of policy learning effectiveness and efficiency on ten robotic manipulation tasks in simulation and five real-worlds. Videos can be found on our project website: https://core-2026.github.io/

Problem

• reward design
• reinforcement learning
• preference alignment
• human intent
• policy optimization

Innovation

• Combined Rewards
• Vision-Language Models
• Preference-Aligned RL
• Formal and Residual Rewards
• Automated Reward Design