TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibility

2025-10-08

AI Summary

Existing video generation models achieve high visual fidelity but frequently produce physically implausible sequences—such as floating objects, teleportation, or acausal deformations—and lack reliable, quantitative benchmarks for assessing physical plausibility. Method: We propose TRAVL, a physics-aware training framework, and ImplausiBench, a novel evaluation benchmark built upon a balanced dataset of physically implausible videos. TRAVL introduces trajectory-aware attention and fine-grained motion encoding to enhance spatiotemporal and causal reasoning in video-language models. Evaluation employs a dual paradigm—human annotation and LLM-based adjudication—to mitigate linguistic bias. Contribution/Results: Our approach achieves human-level accuracy in detecting physical violations on ImplausiBench, marking the first systematic improvement in multimodal models’ visual-temporal discrimination of physical plausibility. This work establishes a new standard for trustworthy evaluation of video generation systems.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Despite impressive visual fidelity, modern video generative models frequently produce sequences that violate intuitive physical laws, such as objects floating, teleporting, or morphing in ways that defy causality. While humans can easily detect such implausibilities, there remains no robust method for quantitatively assessing physical realism in video. In this work, we explore whether Video-Language Models (VLMs) can be trained to serve as reliable judges of physical plausibility. We find that existing VLMs struggle to identify physics violations, exposing fundamental limitations in their temporal and causal reasoning. To address this, we introduce TRAVL, a fine-tuning recipe that combines a balanced training dataset with a trajectory-aware attention module to improve motion encoding and discrimination in VLMs. To evaluate physical reasoning more rigorously, we propose ImplausiBench, a benchmark of 300 videos (150 real, 150 generated) that removes linguistic biases and isolates visual-temporal understanding. Performance is reported both with gold-standard human judgments and stricter LLM-as-judge metrics. Together, TRAVL and ImplausiBench offer a unified framework for probing and improving physical plausibility in multimodal models, shedding light on a challenging and underexplored aspect of visual-temporal understanding.

Problem

• Quantitatively assessing physical realism in generated videos
• Improving Video-Language Models' temporal and causal reasoning
• Creating benchmark for physics plausibility without linguistic biases

Innovation

• Fine-tunes VLMs with balanced dataset training
• Adds trajectory-aware attention for motion encoding
• Introduces ImplausiBench benchmark for rigorous evaluation