CouCE: A Unified Causal Framework for Debiased Deep Metric Learning

2026-06-29

AI Summary

This work addresses the vulnerability of deep metric learning to spurious correlations—specifically, background-induced confounding and foreground non-semantic perturbations—which mislead models into relying on shortcuts rather than intrinsic semantic similarity for zero-shot generalization. To tackle these structurally distinct confounding pathways simultaneously, the authors propose CouCE, a unified causal framework that integrates Orthogonal Dictionary Backdoor Adjustment (ODBA) and Multi-Scale Random Causal Intervention (MSRCI). CouCE can be seamlessly incorporated into any proxy-based loss without altering the inference architecture, leveraging symmetric KL invariance constraints and soft orthogonality regularization to effectively debias representations. Extensive experiments demonstrate that CouCE achieves state-of-the-art performance on CUB-200-2011, Cars-196, and Stanford Online Products, significantly enhancing model robustness and generalization capability.

Abstract

Deep Metric Learning (DML) often struggles with zero-shot generalization because standard objectives inherently capture what co-occurs rather than what causes similarity. Consequently, DML models are vulnerable to shortcut learning driven by two structurally distinct confounders: background spurious correlations (which create backdoor paths via scene context) and foreground nuisance perturbations (which inject non-semantic variations like pose or illumination). Although existing methods have proposed targeted solutions for each pathway individually, none can simultaneously address both due to their fundamentally distinct causal roles. To bridge this gap, we propose the Counterfactual Causal Embedding (CouCE), a unified causal framework that explicitly models and neutralizes both confounders. Specifically, we introduce Orthogonal Dictionary-Based Backdoor Adjustment (ODBA), which isolates spurious background patterns into a variance-gated dictionary and stably disentangles them from the learned embeddings via soft orthogonal regularization. Simultaneously, we propose Multi-Scale Randomized Causal Intervention (MSRCI) to enforce causal invariance against foreground nuisances through multi-scale Fourier amplitude randomization and a symmetric KL invariance constraint. Notably, CouCE seamlessly integrates with any proxy-based loss, incurring modest training overhead without requiring architectural modifications during inference. Extensive experiments on CUB-200-2011, Cars-196, and Stanford Online Products demonstrate that CouCE consistently achieves state-of-the-art performance, providing a principled and robust solution for debiased DML.

Problem

• Deep Metric Learning
• Zero-shot Generalization
• Confounding Bias
• Shortcut Learning
• Causal Invariance

Innovation

• Causal Inference
• Deep Metric Learning
• Debiasing
• Counterfactual Intervention
• Confounding Disentanglement