Benchmarking Single-Factor Physical Video-to-Audio Generation

2026-05-28

AI Summary

This work addresses the overreliance of existing video-to-audio generation models on perceptual realism while neglecting consistency with physical laws. To bridge this gap, we introduce FlatSounds—the first benchmark specifically designed for evaluating physical causal reasoning in audio generation. Leveraging controlled counterfactual video pairs and intra-video consistency analysis, FlatSounds quantifies the accuracy of generated audio in terms of physical plausibility and temporal alignment. Human preference studies further validate the benchmark’s assessments. Our findings reveal that although state-of-the-art models enhance semantic and physical fidelity through text conditioning, they often compromise temporal coherence. Importantly, the proposed physical metrics exhibit strong agreement with human judgments, underscoring the validity and necessity of FlatSounds as a rigorous evaluation framework.

Abstract

Generative video-to-audio (V2A) models produce highly plausible soundtracks, but it remains unclear whether they capture the underlying physical processes. Existing evaluations emphasize perceptual realism and overlook physical correctness under controlled interventions. In this paper, we introduce FlatSounds, a benchmark that audits the physical reasoning of V2A models through: 1) controlled counterfactual pairs in which a single physical factor is varied, and 2) single-video pattern tests that probe internal consistency and directional trends. These settings test whether the generated audio correctly reflects specific physical properties and timings. Our evaluation of state-of-the-art models reveals a consistent trade-off: models rely more on text captions than the visual stream to infer physics and semantics. Captions generally improve physical and semantic accuracy, but paradoxically degrade temporal alignment. Our results highlight the need to move beyond audio quality toward learning physical processes directly from pixels. Finally, we find that our physics-based metrics correlate strongly with human preference tests on our own data. Project webpage: https://research.nvidia.com/labs/cosmos-lab/flatsounds/

Problem

• video-to-audio generation
• physical reasoning
• benchmarking
• counterfactual evaluation
• temporal alignment

Innovation

• video-to-audio generation
• physical reasoning
• counterfactual benchmarking
• temporal alignment
• multimodal grounding