Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Models

2026-07-10

AI Summary

This work addresses the limitations of existing audio-visual generation evaluation methods, which are confined to low-level temporal alignment, struggle to assess structural correctness and cross-modal causal consistency, and rely heavily on human annotations. To overcome these issues, the authors propose the first reference-free synchronization evaluation framework. They construct SynthSync, a human relative preference dataset, and leverage an Omni-LLM with continuous latent projection to map pairwise judgments into globally consistent absolute synchronization scores. Furthermore, they introduce real-valued Group Relative Policy Optimization (ℝ-GRPO) to explicitly model causal structure. The proposed approach achieves state-of-the-art alignment with human preferences and establishes the first standardized benchmark for evaluating audio-visual generation with respect to visual causal consistency.

Abstract

As audio-visual generative models evolve into world simulators, cross-modal synchronization stands as a critical proxy for assessing the consistency of world dynamics and causality in generated content. However, existing evaluation metrics presume structural correctness, reducing synchronization to mere temporal alignment. Consequently, they fail on generative outputs, especially when exhibiting structural hallucinations and asymmetric cross-modal relations, which currently \textbf{mandate expert human annotation to assess synchronization.} This dependency introduces a critical paradox: \emph{human evaluators rely on relative, reference-dependent comparisons, whereas automated metrics require reference-free, absolute scalars.} We resolve this paradox by proposing a framework that distills relative human perception into a continuous, globally consistent metric. First, we introduce SynthSync, a dataset of generative failures ranked via pairwise human annotations. Second, we adapt the Omni-LLM equipped with a continuous latent projection to translate relative human rankings into continuous absolute values. Third, we propose Real-Valued Group Relative Policy Optimization ($\mathbb{R}$-GRPO) to internalize the global causal structure of synchronization via listwise score distributions. Empirically, our metric achieves state-of-the-art human preference alignment. We leverage this estimator to establish a standardized benchmark, advancing AV-Gen assessment from low-level signal correlation to visually grounded causality.

Problem

• cross-modal synchronization
• generative audio-visual models
• reference-free evaluation
• structural hallucinations
• human annotation

Innovation

• reference-free evaluation
• cross-modal synchronization
• Omni-LLM adaptation
• real-valued group relative policy optimization
• audio-visual generative models