CAUSAL3D: A Comprehensive Benchmark for Causal Learning from Visual Data

2025-03-06

AI Summary

A lack of standardized benchmarks for evaluating models’ ability to infer latent causal relationships from complex visual data has hindered progress in causal vision learning. Method: We introduce the first comprehensive benchmark integrating 3D visual scenes with structured causal tables, featuring a novel scene–table alignment design that spans multiple viewpoints, diverse backgrounds, and 19 distinct causal graph structures—enabling evaluation across varying levels of causal complexity. Our framework unifies 3D generative rendering, causal graph modeling, causal discovery algorithms, causal representation learning, and vision-language/ large language model (VLM/LLM) reasoning pipelines. Contribution/Results: Experiments reveal substantial performance degradation of state-of-the-art models under prior-free, complex causal scenarios. This work fills a critical gap in causal visual reasoning evaluation by providing a reproducible, extensible, and modular quantitative benchmark—establishing the first principled foundation for rigorous, fine-grained assessment of causal inference in vision.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

True intelligence hinges on the ability to uncover and leverage hidden causal relations. Despite significant progress in AI and computer vision (CV), there remains a lack of benchmarks for assessing models' abilities to infer latent causality from complex visual data. In this paper, we introduce extsc{ extbf{Causal3D}}, a novel and comprehensive benchmark that integrates structured data (tables) with corresponding visual representations (images) to evaluate causal reasoning. Designed within a systematic framework, Causal3D comprises 19 3D-scene datasets capturing diverse causal relations, views, and backgrounds, enabling evaluations across scenes of varying complexity. We assess multiple state-of-the-art methods, including classical causal discovery, causal representation learning, and large/vision-language models (LLMs/VLMs). Our experiments show that as causal structures grow more complex without prior knowledge, performance declines significantly, highlighting the challenges even advanced methods face in complex causal scenarios. Causal3D serves as a vital resource for advancing causal reasoning in CV and fostering trustworthy AI in critical domains.

Problem

• Lack of benchmarks for causal inference in visual data.
• Need to evaluate models on complex causal relations in 3D scenes.
• Challenges in causal reasoning without prior knowledge.

Innovation

• Integrates structured data with visual representations
• Includes 19 diverse 3D-scene datasets
• Evaluates causal reasoning across varying complexities