Dual-Branch Cross-Projection Debiasing through Diffusion-based Disentanglement

2026-06-23

AI Summary

This work addresses the vulnerability of foundation models to spurious correlations between labels and non-causal attributes in biased data, which leads to poor generalization on minority groups and hinders identification of semantically aligned bias factors when group labels are unavailable. To tackle this, the authors propose a Dual-branch Cross-projection Debiasing (DCD) framework that, for the first time, integrates diffusion models for unsupervised bias concept discovery. Through Confidence-guided Bias Concept Mining (CBCM), DCD obtains semantically coherent bias representations and explicitly disentangles target and spurious attributes in separate branches via cross-nullspace projection, thereby removing misleading information. This approach overcomes the limitations of single-branch architectures with shared feature spaces and achieves state-of-the-art worst-group accuracy under the challenging setting without group annotations, fine-tuning at most only 0.22% of model parameters across four benchmark datasets.

Abstract

Foundation models trained on biased datasets often rely on spurious correlations between target labels and non-causal attributes, resulting in poor generalization on minority groups. Bias mitigation remains challenging due to two fundamental issues. First, when group labels are unavailable, existing group-unsupervised methods typically infer spurious attributes implicitly from model behavior, making it difficult to identify spurious factors that are semantically aligned with real-world biases. Second, even with pseudo spurious supervision, most existing debiasing methods follow a single-branch design that operates within a single shared feature space, where target and spurious attributes are intrinsically entangled. To address the first challenge, we introduce Confidence-guided Bias Concept Mining (CBCM), which leverages diffusion-disentangled, semantically grounded concept representations to identify reliable spurious attributes without attribute annotations. To address the second challenge, we propose Dual-branch Cross-projection Debiasing (DCD), a prompt-tuning framework that separates target and spurious representations into two branches and explicitly removes spurious information through cross null-space projection while preserving target-relevant semantics. Extensive experiments on four benchmark datasets show that our method achieves state-of-the-art worst group accuracy among group-unsupervised approaches, while tuning at most 0.22% of the model parameters. The source code is available in the supplementary materials.

Problem

• bias mitigation
• spurious correlations
• group-unsupervised debiasing
• feature disentanglement
• foundation models

Innovation

• diffusion-based disentanglement
• dual-branch architecture
• cross null-space projection
• group-unsupervised debiasing
• spurious correlation mitigation