Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Models

2026-05-08

AI Summary

Reward modeling for video understanding is hindered by the scarcity of high-quality preference data and reliable evaluation benchmarks. This work proposes a unified framework that introduces VURB, the first comprehensive benchmark for video reward modeling encompassing diverse tasks and long reasoning chains, alongside VUP-35K, a large-scale preference dataset generated through a fully automated pipeline. Building upon this foundation, we develop both a discriminative reward model (VideoDRM) and a generative reward model (VideoGRM). Experimental results demonstrate that both models achieve state-of-the-art performance on VURB and VideoRewardBench, significantly enhancing reasoning capabilities and generation quality in best-of-N settings.

Abstract

Multimodal reward models have advanced substantially in text and image domains, yet progress in video understanding reward modeling remains severely limited by the lack of robust evaluation benchmarks and high-quality preference data. To address this, we propose a unified framework spanning benchmark design, data construction, and reward model training. We introduce Video Understanding Reward Bench (VURB), a benchmark featuring 2,100 preference pairs with long chain-of-thought reasoning traces (averaging 1,143 tokens) and majority voting evaluation across general, long, and reasoning-oriented video tasks. We further construct Video Understanding Preference Dataset (VUP-35K) via a fully automated pipeline, providing large-scale high-quality supervision for video reward training. Building on the data, we train VideoDRM and VideoGRM, a discriminative and a generative reward model, both achieving state-of-the-art performance on VURB and VideoRewardBench. Further analysis confirms that VUP-35K enhances both reward performance and model reasoning capability, while VideoDRM and VideoGRM yield significant gains under best-of-$N$ test-time scaling.

Problem

• video understanding
• reward modeling
• evaluation benchmark
• preference data
• multimodal reward models

Innovation

• video reward modeling
• preference dataset
• automated data pipeline
• chain-of-thought reasoning
• multimodal benchmark