AspectCLIP: Optimizing CLIP Representation Space via Aspect-Guided Consistency Regularization

2026-07-15

AI Summary

This work addresses the limitation of existing CLIP-based methods that overlook the semantic asymmetry between images and text in global consistency regularization, often leading to distorted representations. To mitigate this issue, the authors propose an aspect-oriented guided consistency regularization mechanism. Specifically, they first construct semantically coherent sample groups based on textual similarity and enforce cycle-consistency constraints within each group to effectively model one-to-many semantic correspondences. Between groups, only prototype-level contrastive learning is applied to prevent cross-semantic interference. This approach enhances both the structural organization and semantic fidelity of the multimodal representation space, yielding significant performance improvements over baseline models across multiple downstream tasks.

Abstract

Contrastive Language-Image Pretraining learns a shared representation space through large-scale contrastive learning. However, existing methods that enforce global consistency regularization overlook a key challenge: the inherent information asymmetry between images and text: captions typically describe only one specific aspect of an image, thus images with similar visual content can be paired with completely divergent textual content and semantic information. Consequently, global regularizers inadvertently impose constraints between visually similar images whose captions describe divergent aspects, introducing semantic distortion into the representation space. We propose AspectCLIP, a framework that reformulates consistency regularization to respect this one-to-many structure. AspectCLIP first partitions training samples into attribute clusters based on textual similarity to identify aspect-coherent groups, then applies full cyclic consistency within each cluster while restricting cross-cluster regularization to prototype-level comparisons. This aspect-guided regularization enforces strict geometric alignment only when images and texts describe a consistent facet, while allowing flexibility across divergent aspects. Extensive experiments on downstream tasks demonstrate that AspectCLIP consistently outperforms traditional methods and achieves a more structured representation space.

Problem

• information asymmetry
• aspect inconsistency
• representation distortion
• contrastive learning
• image-text alignment

Innovation

• Aspect-Guided Regularization
• Information Asymmetry
• Attribute Clustering
• Cyclic Consistency
• Representation Space Optimization