Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events

2024-12-07

arXiv.org

AI Summary

Vision-language models (VLMs) exhibit limited abductive and defeasible reasoning capabilities when confronted with atypical, real-world video events—highlighting a critical gap in causal understanding beyond statistical pattern recognition. Method: We introduce BlackSwanSuite, the first benchmark dedicated to non-canonical video events, designed to rigorously evaluate dynamic belief revision and implicit causal inference under constrained visual inputs and hypothesis-updating tasks. Contribution/Results: BlackSwanSuite features a multimodal, formalized framework integrating abductive and defeasible logic, with 3,800+ multiple-choice, 4,900+ open-ended generation, and 6,700+ true/false questions across 1,655 rare-event videos. Evaluation on state-of-the-art VLMs—including GPT-4o, Gemini 1.5 Pro, and LLaVA-Video—reveals up to a 32% human–model performance gap. The benchmark is fully open-sourced, accompanied by a public leaderboard, to advance robust causal reasoning research in VLMs.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The commonsense reasoning capabilities of vision-language models (VLMs), especially in abductive reasoning and defeasible reasoning, remain poorly understood. Most benchmarks focus on typical visual scenarios, making it difficult to discern whether model performance stems from keen perception and reasoning skills, or reliance on pure statistical recall. We argue that by focusing on atypical events in videos, clearer insights can be gained on the core capabilities of VLMs. Explaining and understanding such out-of-distribution events requires models to extend beyond basic pattern recognition and regurgitation of their prior knowledge. To this end, we introduce BlackSwanSuite, a benchmark for evaluating VLMs' ability to reason about unexpected events through abductive and defeasible tasks. Our tasks artificially limit the amount of visual information provided to models while questioning them about hidden unexpected events, or provide new visual information that could change an existing hypothesis about the event. We curate a comprehensive benchmark suite comprising over 3,800 MCQ, 4,900 generative and 6,700 yes/no questions, spanning 1,655 videos. After extensively evaluating various state-of-the-art VLMs, including GPT-4o and Gemini 1.5 Pro, as well as open-source VLMs such as LLaVA-Video, we find significant performance gaps of up to 32% from humans on these tasks. Our findings reveal key limitations in current VLMs, emphasizing the need for enhanced model architectures and training strategies. Our data and leaderboard is available at blackswan.cs.ubc.ca.

Problem

• Evaluating VLMs' reasoning in unpredictable video events
• Assessing abductive and defeasible reasoning beyond typical scenarios
• Identifying performance gaps between VLMs and humans

Innovation

• BlackSwanSuite benchmark for unexpected events
• Abductive and defeasible reasoning tasks
• Limited visual information for evaluation