MSCI: Addressing CLIP's Inherent Limitations for Compositional Zero-Shot Learning

2025-05-15

AI Summary

To address CLIP’s limitation in modeling fine-grained local visual features for Compositional Zero-Shot Learning (CZSL), this paper proposes a multi-stage cross-modal interaction framework. The method introduces an adaptive high-low-level visual feature aggregator that fuses CLIP’s intermediate-layer low-level local textures with high-level semantic representations, and incorporates dual-granularity dynamic attention—operating at both compositional and elemental levels—to enable staged text–vision alignment across abstraction levels. Crucially, the approach preserves CLIP’s frozen pre-trained weights and constructs only lightweight interaction modules atop its intermediate features. Evaluated on three standard CZSL benchmarks—MIT-States, UT-Zappos, and CGQA—the method achieves significant improvements over state-of-the-art methods, particularly in generalizing to unseen state–object compositions. These results empirically validate the effectiveness of joint local–global modeling for disentangling compositional semantics.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen state-object combinations by leveraging known combinations. Existing studies basically rely on the cross-modal alignment capabilities of CLIP but tend to overlook its limitations in capturing fine-grained local features, which arise from its architectural and training paradigm. To address this issue, we propose a Multi-Stage Cross-modal Interaction (MSCI) model that effectively explores and utilizes intermediate-layer information from CLIP's visual encoder. Specifically, we design two self-adaptive aggregators to extract local information from low-level visual features and integrate global information from high-level visual features, respectively. These key information are progressively incorporated into textual representations through a stage-by-stage interaction mechanism, significantly enhancing the model's perception capability for fine-grained local visual information. Additionally, MSCI dynamically adjusts the attention weights between global and local visual information based on different combinations, as well as different elements within the same combination, allowing it to flexibly adapt to diverse scenarios. Experiments on three widely used datasets fully validate the effectiveness and superiority of the proposed model. Data and code are available at https://github.com/ltpwy/MSCI.

Problem

• Overcoming CLIP's limitations in fine-grained local feature capture
• Enhancing compositional zero-shot learning via multi-stage cross-modal interaction
• Improving recognition of unseen state-object combinations dynamically

Innovation

• Multi-Stage Cross-modal Interaction (MSCI) model
• Self-adaptive aggregators for local and global features
• Dynamic attention weights adjustment for diverse scenarios