InfluenceField: A Differentiable Field with Interventionally Identifiable Causal Structure for Multimodal World Modeling

2026-09-07

AI Summary

为解决多模态大模型中视觉干预效果预测问题，提出InfluenceField方法，在视觉编码器和语言解码器间插入可微且具有因果结构的场，通过联合训练优化多个目标。

Abstract

Multimodal large language models often capture visual-linguistic correlations but struggle to predict how local visual interventions propagate and affect downstream answers. We introduce InfluenceField, an intervention-aware latent field inserted between the visual encoder and language decoder. It lifts patch features into a continuous spatial representation, propagates directed influence over multiple steps, and predicts local intervention effects through a shared transition operator. Training jointly optimizes language modeling, cross-environment invariance, counterfactual rollout supervision, and structural regularization. For a nonlinear finite-basis population model, we show that target-aligned interventional supervision, together with a one-step separation condition on the transition, restricts admissible representations to within-location reparameterizations, so that the directed dependency graph of the full transition is recovered exactly. A linear specialization gives an exact partial-coverage characterization and a finite-loss stability bound, and the field analysis derives the spatial profile of coefficient interventions together with a shared-channel calibration result. On CausalVQA, InfluenceField improves overall accuracy over its backbone by 13.1 percentage points, with the largest gains on the planning and hypothetical categories. Capacity-matched baselines and structural controls attribute the gains in robustness and factual-counterfactual consistency to the causal objectives rather than to added capacity.

Problem

• multimodal large language models
• visual-linguistic correlations
• local visual interventions
• downstream answers

Innovation

• InfluenceField
• intervention-aware
• cross-environment invariance
• counterfactual rollout supervision
• causal structure