A Causal Adjustment Module for Debiasing Scene Graph Generation.

2025-01-30

IEEE Transactions on Pattern Analysis and Machine Intelligence

AI Summary

Existing scene graph generation (SGG) debiasing methods primarily address the long-tailed distribution of *relationships*, overlooking deeper bias sources arising from skewed distributions at the *object* and *object-pair* levels. This work pioneers the integration of causal mediation analysis into SGG, proposing a three-tier causal chain model—“Object → Object Pair → Relationship” (MCCM)—with co-occurrence statistics as the mediating variable. We design a Causal Adjustment Module (CAModule) to perform counterfactual debiasing by disentangling direct and indirect causal effects. The method enables zero-shot relationship generalization and is plug-and-play compatible with mainstream SGG backbone architectures. Evaluated on multiple benchmarks, it achieves state-of-the-art mean recall while substantially improving zero-shot recall (+5.2%–9.8%). These results empirically validate that joint modeling of multi-level distributional biases effectively mitigates systemic bias in SGG.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While recent debiasing methods for Scene Graph Generation (SGG) have shown impressive performance, these efforts often attribute model bias solely to the long-tail distribution of relationships, overlooking the more profound causes stemming from skewed object and object pair distributions. In this paper, we employ causal inference techniques to model the causality among these observed skewed distributions. Our insight lies in the ability of causal inference to capture the unobservable causal effects between complex distributions, which is crucial for tracing the roots of model bias. Specifically, we introduce the Mediator-based Causal Chain Model (MCCM), which, in addition to modeling causality among objects, object pairs, and relationships, incorporates mediator variables, i.e., cooccurrence distribution, for complementing the causality. Following this, we propose the Causal Adjustment Module (CAModule) to estimate the modeled causal structure, using variables from MCCM as inputs to produce a set of adjustment factors aimed at correcting biased model predictions. Moreover, our method enables the composition of zero-shot relationships, thereby enhancing the model's ability to recognize such relationships. Experiments conducted across various SGG backbones and popular benchmarks demonstrate that CAModule achieves state-of-the-art mean recall rates, with significant improvements also observed on the challenging zero-shot recall rate metric.

Problem

• Addresses model bias in Scene Graph Generation beyond long-tail relationships
• Models causality among skewed object and object pair distributions
• Enhances zero-shot relationship recognition via causal adjustment

Innovation

• Causal inference models skewed distributions
• Mediator-based Causal Chain Model enhances causality
• Causal Adjustment Module corrects biased predictions