CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation

2025-02-27

AI Summary

This work identifies structural biases in CLIP’s multimodal representations under multi-object scenarios: the text encoder exhibits strong positional bias toward the first-mentioned object, while the image encoder shows pronounced size bias toward larger objects. To quantify these instabilities under object ordering and scale variations, the authors introduce ComCO—the first fine-grained, multi-object evaluation benchmark for CLIP. Through statistical analysis of LAION data, training dynamics tracking, and joint text-image perturbation experiments, they attribute the biases to imbalances in training data distribution and optimization dynamics. Crucially, they demonstrate that these biases propagate to downstream generative models, notably Stable Diffusion. The contributions are threefold: (1) a systematic diagnostic framework and the reproducible ComCO benchmark; (2) a mechanistic explanation of CLIP’s limitations in multi-object understanding; and (3) theoretical and empirical foundations for developing robust multi-object representations and controllable multimodal generation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Contrastive Language-Image Pre-training (CLIP) models excel in zero-shot classification, yet face challenges in complex multi-object scenarios. This study offers a comprehensive analysis of CLIP's limitations in these contexts using a specialized dataset, ComCO, designed to evaluate CLIP's encoders in diverse multi-object scenarios. Our findings reveal significant biases: the text encoder prioritizes first-mentioned objects, and the image encoder favors larger objects. Through retrieval and classification tasks, we quantify these biases across multiple CLIP variants and trace their origins to CLIP's training process, supported by analyses of the LAION dataset and training progression. Our image-text matching experiments show substantial performance drops when object size or token order changes, underscoring CLIP's instability with rephrased but semantically similar captions. Extending this to longer captions and text-to-image models like Stable Diffusion, we demonstrate how prompt order influences object prominence in generated images. For more details and access to our dataset and analysis code, visit our project repository: https://clip-analysis.github.io.

Problem

• Analyzes CLIP's multi-object representation biases
• Quantifies biases in CLIP's text and image encoders
• Explores prompt order impact on object prominence

Innovation

• Analyzes CLIP's multi-object representation biases
• Uses ComCO dataset for evaluation
• Investigates prompt order impact on image generation