Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning

2025-03-23

AI Summary

Large Vision-Language Models (LVLMs) heavily rely on labor-intensive human preference annotations and task-specific reward models for alignment. Method: This paper proposes a human-annotation-free, vision-guided reinforcement learning framework. Its core innovations are: (1) the first vision-feedback-driven R1-style RL algorithm, where raw images serve as intrinsic supervisory signals; (2) a generalizable, multi-dimensional task-logic reward function jointly measuring accuracy, consistency, and instruction adherence; and (3) a dynamic progressive rule optimization mechanism to effectively mitigate reward hacking. Results: Evaluated on a 7B LVLM, the method achieves comprehensive performance gains—key metrics improve by up to 50%—and substantially outperforms state-of-the-art models with ten times its parameter count.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Vision-Language Models (LVLMs) typically follow a two-stage training paradigm-pretraining and supervised fine-tuning. Recently, preference optimization, derived from the language domain, has emerged as an effective post-training reinforcement strategy to enhance capabilities of LVLMs. However, constructing high-quality human-annotated preference data and developing robust reward models to mimic these preferences are both costly and challenging. Motivated by this observation, we propose Vision-R1, a novel vision-guided R1-like reinforcement learning algorithm for LVLMs that rewards models with definitive vision feedback. It only leverages curated instruction data, eliminating the need for specialized reward models and handcrafted preference datasets. We incorporate a criterion-driven reward function that further integrates multi-dimensional feedback to evaluate model completions comprehensively based on the vision task logic. Furthermore, we introduce a progressive rule refinement strategy that dynamically adjusts the reward criteria during training, enabling continuous model improvement and mitigating reward hacking. Extensive experiments on both in-distribution and out-of-distribution benchmarks demonstrate that fine-tuning the 7B LVLMs with Vision-R1 achieves consistent performance gains, with even up to 50% improvement and surpassing the state-of-the-art 10x size model.

Problem

• Eliminates need for human-annotated preference data
• Develops vision-guided reinforcement learning for LVLMs
• Improves model performance without specialized reward models

Innovation

• Vision-guided R1-like reinforcement learning algorithm
• Criterion-driven reward function with multi-dimensional feedback
• Progressive rule refinement strategy for dynamic adjustment