Learning Visual Composition through Improved Semantic Guidance

2024-12-19

arXiv.org

AI Summary

Existing vision representation learning excels at modeling discrete objects but struggles with compositional reasoning—largely constrained by contrastive learning paradigms that treat images as “bags of words.” Method: We propose a model-agnostic, high-quality semantic-guided caption engineering framework—requiring no architectural modifications—to substantially enhance standard CLIP’s compositional reasoning capability. Our approach includes (i) fine-grained concept-aligned caption augmentation and (ii) construction of DOCCI-derived compositional understanding benchmarks. Contribution/Results: Experiments demonstrate dramatic gains: performance on compositional understanding tasks improves from near-random to state-of-the-art—surpassing all specialized compositional models—and achieves strong cross-domain image retrieval generalization. Crucially, this work provides the first empirical evidence that high-quality weakly supervised textual signals inherently encode sufficient compositional semantics, challenging the prevailing assumption that novel architectures are necessary for compositional reasoning.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Visual imagery does not consist of solitary objects, but instead reflects the composition of a multitude of fluid concepts. While there have been great advances in visual representation learning, such advances have focused on building better representations for a small number of discrete objects bereft of an understanding of how these objects are interacting. One can observe this limitation in representations learned through captions or contrastive learning -- where the learned model treats an image essentially as a bag of words. Several works have attempted to address this limitation through the development of bespoke learned architectures to directly address the shortcomings in compositional learning. In this work, we focus on simple, and scalable approaches. In particular, we demonstrate that by substantially improving weakly labeled data, i.e. captions, we can vastly improve the performance of standard contrastive learning approaches. Previous CLIP models achieved near chance rate on challenging tasks probing compositional learning. However, our simple approach boosts performance of CLIP substantially and surpasses all bespoke architectures. Furthermore, we showcase our results on a relatively new captioning benchmark derived from DOCCI. We demonstrate through a series of ablations that a standard CLIP model trained with enhanced data may demonstrate impressive performance on image retrieval tasks.

Problem

• Improving visual composition understanding in representation learning
• Enhancing weakly labeled data for better contrastive learning
• Boosting CLIP model performance on compositional tasks

Innovation

• Improving weakly labeled data for better performance
• Enhancing standard contrastive learning approaches
• Using enhanced data to boost CLIP model performance