Through the PRISM: Preference Representation in Intermediate States of Video Diffusion Models

2026-06-18

AI Summary

This work addresses the limitations of conventional pixel-based video reward models, which are decoupled from the diffusion process and rely on computationally expensive VAE decoding. The authors propose PRISM, the first method to demonstrate that video diffusion models can effectively discriminate generation quality directly within noisy intermediate latent representations. PRISM leverages a frozen diffusion backbone augmented with a lightweight query aggregation head to extract preference signals directly from the noisy latent space, enabling efficient preference scoring and early candidate filtering. While keeping the backbone frozen, PRISM supports optional fine-tuning with substantially reduced computational overhead. It achieves state-of-the-art performance in preference accuracy and enhances final video quality through effective early-stage selection.

Abstract

Evaluating video generation with clean, pixel-based reward models disconnects evaluation from the noisy diffusion process and incurs massive VAE decoding costs. In this paper, we challenge this paradigm by asking a fundamental question: Can a powerful video generator inherently discriminate preferences directly from noisy latents? To answer this, we introduce \textbf{PRISM} (\textbf{P}reference \textbf{R}epresentation in \textbf{I}ntermediate \textbf{S}tates of Diffusion \textbf{M}odels). PRISM employs a lightweight Query-based Aggregation head with a frozen video diffusion backbone to decode preference signals from noisy latents. Surprisingly, PRISM not only achieves SOTA preference accuracy but also unlocks strong noise-robustness, which enables early-stage Best-of-$N$ sampling. This allows for filtering suboptimal candidates at the very beginning of denoising, drastically reducing computation while boosting video quality. We also reveal a strong positive correlation between a backbone's generative performance and its inherent evaluative power, enabling self-improving video backbones.

Problem

• video generation
• preference evaluation
• diffusion models
• noisy latents
• VAE decoding

Innovation

• PRISM
• preference representation
• video diffusion models
• noisy latents
• early-stage sampling