From Prompts to Tokens: Internalizing Causal Supervision in Vision-Language Model for Multi-Image Causal Reasoning
Existing vision-language models exhibit fragility in multi-image causal reasoning tasks, struggling with interventional and counterfactual queries and often relying on external textual prompts to inject causal knowledge. This work proposes the first approach that internalizes causal mechanisms directly into the model’s execution pipeline: it constructs multi-image causal graphs encoded as structured causal tokens, introduces a RAMP layer embedded within the LLM decoder to facilitate causal information propagation, and presents a unified training framework, M3S, enabling multi-granular causal supervision at both local and global levels. The method achieves state-of-the-art performance, attaining 54.4% accuracy (+21.2%) on the CausalVLBench interventional tasks, improving accuracy to 49.0% on Causal3D, and significantly boosting the F1 score for causal structure learning to 75.1% (+41.7%).