VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning

2026-07-01

AI Summary

This work addresses the challenge of reward function design in reinforcement learning within open-ended environments, where goals are often abstract and difficult to quantify. The authors propose a novel reward learning framework leveraging vision-language models (VLMs) that uniquely combines absolute rewards—evaluating individual states—with relative rewards derived from comparing consecutive observations to assess task progress. By integrating state-level evaluation stability and the robustness of comparative supervision, the method jointly optimizes preference labeling, absolute reward modeling, and relative reward inference through multi-task reinforcement learning. Evaluated on visually complex, long-horizon benchmark tasks such as Minecraft, the approach significantly outperforms existing VLM-based reward learning methods.

Abstract

Designing effective reward functions remains a major challenge in reinforcement learning (RL), particularly in open-ended environments where task goals are abstract and difficult to quantify. In this work, we present VLM-AR3L, a framework that leverages Vision-Language Models (VLMs) to provide both absolute and relative rewards for RL. VLM-AR3L interprets an agent's visual observations in the context of a natural language task goal, and learns both absolute and relative rewards from VLM-generated preference labels. The absolute reward model predicts scalar evaluations for individual states, while the relative reward model compares consecutive observations to infer progress or regression toward the task goal. Their integration combines the stability of state-based evaluation with the robustness of comparative supervision. We evaluate VLM-AR3L across benchmarks spanning classic control, manipulation, and open-world embodied tasks, with a particular focus on Minecraft given its visual complexity and long-horizon decision-making requirements. Experimental results show that VLM-AR3L consistently outperforms prior VLM-based reward learning methods.

Problem

• reward function
• reinforcement learning
• open-ended environments
• task goals
• Vision-Language Models

Innovation

• Vision-Language Models
• Absolute Reward
• Relative Reward
• Reinforcement Learning
• Reward Learning