Trimming the Long-Tail of Visual World Modeling Evaluation

2026-06-23

AI Summary

While existing visual world models can handle common physical interactions, they exhibit limited generalization in rare or unconventional scenarios due to a lack of deep understanding of physical principles. To address this gap, this work proposes Tailor-Bench—the first structured evaluation benchmark targeting long-tailed physical interactions. Tailor-Bench features three progressively challenging scenario types—Regular, Unconventional, and Impossible—and incorporates both prediction-generation and description-generation settings to systematically assess models’ physical commonsense reasoning. The benchmark explicitly distinguishes between attribute-compatible and attribute-violating tool-task pairings and employs a unified protocol with hybrid evaluation metrics. Experiments reveal a significant performance drop as scenario unconventionality increases: image-based models struggle to model state changes, while video-based models suffer from temporal inconsistencies, indicating their reliance on superficial visual patterns rather than genuine physical mechanisms.

Abstract

Physical interactions follow a long-tailed distribution: a set of common and regular interactions dominates human experience and visual data, while a broad spectrum of rare and irregular interactions remains underrepresented. Although recent visual world models, including image and video generation models, achieve impressive realism on existing benchmarks, they primarily focus on simulating common physical interactions. This raises a central question: Do current visual world models internalize and generalize physical principles? In this work, we introduce Tailor-Bench, a benchmark that challenges world models to simulate irregular physical interactions. To enable systematic evaluation, we design three scenario modes that progressively challenge model reasoning: Regular scenarios reflect common tool-task pairs, Unconventional scenarios replace conventional tools with attribute-compatible substitutes to test affordance generalization, and Impossible scenarios introduce attribute-violating tools to probe constraint awareness. Additionally, we design two complementary settings under a unified evaluation protocol: predictive generation requires inferring outcomes without guidance, while descriptive generation specifies the target outcome for faithful realization. Our experimental results reveal a clear long-tail gap in physical world modeling: performance degrades from Regular to Unconventional and Impossible scenarios, indicating limited generalization beyond common interactions. Failure analysis further shows that models rely on superficial visual patterns: image models fail to realize correct state changes, while video models further suffer from temporal inconsistencies.

Problem

• long-tailed distribution
• visual world models
• physical interactions
• generalization
• benchmark evaluation

Innovation

• long-tailed evaluation
• physical world modeling
• affordance generalization
• Tailor-Bench
• visual reasoning