BRITE: A Benchmark for Reliable and Interpretable T2V Evaluation on Implausible Scenarios

2026-04-24

AI Summary

This work addresses critical gaps in current text-to-video (T2V) evaluation benchmarks, which often overlook implausible scenes and lack fine-grained assessment of audio-visual consistency. The authors propose the first unified evaluation framework that integrates three key innovations: the incorporation of implausible prompts to test model robustness, the design of fine-grained metrics for audio-visual alignment, and a question-answering-based mechanism to enhance interpretability. By employing a human-in-the-loop benchmark construction pipeline, the framework effectively mitigates hallucination issues inherent in fully automated evaluations. Experiments across five state-of-the-art T2V models reveal that while existing methods can adequately handle static object compositions, their performance significantly degrades on crucial dimensions such as object-action binding and audio-visual synchronization.

Abstract

The rapid advancement of photorealistic Text-to-Video (T2V) generation brings in an urgent need for up-to-date evaluation methods. Existing benchmarks largely overlooked implausible scenarios and do not measure audio-visual alignment. We introduce BRITE, the first framework that unifies (1) implausible prompting, (2) fine-grained assessment of audio-visual consistency, and (3) QA-based interpretable evaluation into a comprehensive T2V benchmark. Unlike fully automated Multimodal LLM-based pipelines, which are prone to hallucination and prompt ambiguity, BRITE guarantees reliability through a rigorous human-in-the-loop protocol for benchmark creation. Evaluating five state-of-the-art models (Sora 2, Veo 3.1, Runway Gen4.5, Pixverse V5.5, and Qwen3Max), we reveal a critical performance gap: while models excel at static object composition, they exhibit significant degradation in object-action binding and audio-visual synchronization. Our framework offers the community a reliable, interpretable benchmark and evaluation framework that can detect and locate limitations in the next generation of T2V models, especially for off-manifold prompts

Problem

• Text-to-Video evaluation
• implausible scenarios
• audio-visual alignment
• benchmark reliability
• off-manifold prompts

Innovation

• implausible prompting
• audio-visual alignment
• interpretable evaluation
• human-in-the-loop benchmarking
• text-to-video generation