CV-DCLR: Causal-Visual Dynamic Label Refinement for Robust Zero-Shot Learning

2026-07-01

AI Summary

This work addresses the challenges in zero-shot learning caused by semantic entanglement, which leads to ambiguous visual representations and spurious correlations. To mitigate these issues, the authors propose CV-DCLR, a novel framework that uniquely integrates causal inference with dynamic label refinement. CV-DCLR employs a dual-stream mutual correction mechanism: one stream models observed data patterns, while the other performs counterfactual interventions to verify the causal necessity of semantic prototypes. An adaptive gating mechanism further enhances causally relevant features and suppresses irrelevant distractions. Extensive experiments on benchmark datasets—CUB, SUN, and AWA2—demonstrate that CV-DCLR significantly outperforms existing methods, particularly in high-entanglement scenarios, where it exhibits robust performance and effectively alleviates the performance degradation typically induced by semantic confusion.

Abstract

Zero-Shot Learning (ZSL) facilitates knowledge transfer via shared semantic spaces. However, a critical bottleneck in this paradigm is Semantic Entanglement, where visual representations are inevitably conflated with visually similar semantic concepts, such as distinguishing the intrinsic traits of a Wolf from the shared features of a Husky. Existing global alignment methods often indiscriminately maximize correlations between visual and semantic modalities, leading models to overfit spurious similarities rather than capturing distinctive class identities. To address this fundamental limitation, we propose the Causal-Visual Dynamic Label Refinement (CV-DCLR) framework. Unlike traditional approaches that rely on superficial visual statistics, CV-DCLR recalibrates visual-semantic associations via a Dual-Stream Mutual Correction Mechanism. This includes a Visual Likelihood Stream to model observational patterns and a Causal Importance Stream that verifies the structural necessity of candidate prototypes through Counterfactual Intervention. Acting as a logical filter, our adaptive gating mechanism dynamically modulates feature responses to amplify genuine causal traits while suppressing visually plausible but structurally irrelevant distractors. Extensive experiments on the CUB, SUN, and AWA2 benchmarks under a rigorous Semantic Entanglement Injection protocol demonstrate that CV-DCLR significantly outperforms state-of-the-art methods in high-ambiguity scenarios. Specifically, while existing models suffer catastrophic degradation under entanglement, our framework maintains robust performance, effectively disentangling true class identities from semantic confounders.

Problem

• Zero-Shot Learning
• Semantic Entanglement
• Visual-Semantic Alignment
• Spurious Correlation
• Class Identity Disentanglement

Innovation

• Causal Inference
• Zero-Shot Learning
• Semantic Disentanglement
• Counterfactual Intervention
• Dynamic Label Refinement