CustomContrast: A Multilevel Contrastive Perspective For Subject-Driven Text-to-Image Customization

2024-09-09

arXiv.org

AI Summary

Existing subject-driven text-to-image customization methods often erroneously conflate extraneous attributes—such as pose and background—with intrinsic subject identity, undermining the simultaneous achievement of visual fidelity and text controllability. To address this, we propose a cross-difference contrastive framework that disentangles intrinsic subject representations from extraneous attributes via hierarchical contrastive learning: (i) cross-modal semantic contrast to align subject identity across modalities; (ii) multi-scale appearance-level feature contrast to suppress background and pose leakage; and (iii) a multimodal feature injection encoder to facilitate stable training. Our method fine-tunes pre-trained diffusion models without architectural modification. Experiments demonstrate substantial improvements: +12.6% in CLIP-Score and +18.3% in Prompt Alignment across multiple benchmarks. Critically, our approach alleviates the overfitting–underfitting trade-off, enhancing both internal consistency (subject coherence across generations) and external discriminability (subject uniqueness across instances).

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Subject-driven text-to-image (T2I) customization has drawn significant interest in academia and industry. This task enables pre-trained models to generate novel images based on unique subjects. Existing studies adopt a self-reconstructive perspective, focusing on capturing all details of a single image, which will misconstrue the specific image's irrelevant attributes (e.g., view, pose, and background) as the subject intrinsic attributes. This misconstruction leads to both overfitting or underfitting of irrelevant and intrinsic attributes of the subject, i.e., these attributes are over-represented or under-represented simultaneously, causing a trade-off between similarity and controllability. In this study, we argue an ideal subject representation can be achieved by a cross-differential perspective, i.e., decoupling subject intrinsic attributes from irrelevant attributes via contrastive learning, which allows the model to focus more on intrinsic attributes through intra-consistency (features of the same subject are spatially closer) and inter-distinctiveness (features of different subjects have distinguished differences). Specifically, we propose CustomContrast, a novel framework, which includes a Multilevel Contrastive Learning (MCL) paradigm and a Multimodal Feature Injection (MFI) Encoder. The MCL paradigm is used to extract intrinsic features of subjects from high-level semantics to low-level appearance through crossmodal semantic contrastive learning and multiscale appearance contrastive learning. To facilitate contrastive learning, we introduce the MFI encoder to capture cross-modal representations. Extensive experiments show the effectiveness of CustomContrast in subject similarity and text controllability.

Problem

• Decoupling intrinsic attributes from irrelevant ones in T2I customization.
• Addressing overfitting and underfitting in subject-driven image generation.
• Enhancing similarity and controllability through contrastive learning.

Innovation

• Multilevel Contrastive Learning for feature extraction
• Multimodal Feature Injection Encoder for cross-modal representations
• Decoupling intrinsic attributes via contrastive learning