Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models

2026-02-07

AI Summary

Existing visual reward models struggle to simultaneously achieve alignment with human preferences, semantic consistency, and efficient inference in complex image editing tasks. This work proposes a unified training framework based on a shared vision-language backbone, which, for the first time, internalizes chain-of-thought reasoning capabilities within a discriminative reward model. By jointly optimizing discriminative preference learning and generative language modeling, the approach unifies semantic understanding and preference alignment. The method achieves state-of-the-art performance on both MMRB2 and EditReward-Bench benchmarks and significantly enhances the stability and sample efficiency of downstream online reinforcement learning.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Reward models are critical for reinforcement learning from human feedback, as they determine the alignment quality and reliability of generative models. For complex tasks such as image editing, reward models are required to capture global semantic consistency and implicit logical constraints beyond local similarity. Existing reward modeling approaches have clear limitations. Discriminative reward models align well with human preferences but struggle with complex semantics due to limited reasoning supervision. Generative reward models offer stronger semantic understanding and reasoning, but they are costly at inference time and difficult to align directly with human preferences. To this end, we propose Joint Reward Modeling (JRM), which jointly optimizes preference learning and language modeling on a shared vision-language backbone. This approach internalizes the semantic and reasoning capabilities of generative models into efficient discriminative representations, enabling fast and accurate evaluation. JRM achieves state-of-the-art results on MMRB2 and EditReward-Bench, and significantly improves stability and performance in downstream online reinforcement learning. These results show that joint training effectively bridges efficiency and semantic understanding in reward modeling.

Problem

• reward modeling
• visual reward models
• semantic consistency
• human preference alignment
• reasoning supervision

Innovation

• Joint Reward Modeling
• Chain-of-Thought Reasoning
• Vision-Language Backbone
• Preference Learning
• Efficient Reward Models