Exploring Palette based Color Guidance in Diffusion Models

2025-08-12

AI Summary

Current text-to-image (T2I) diffusion models struggle to achieve global, consistent color control—particularly in backgrounds and non-salient regions. To address this, we propose a palette-guided mechanism that incorporates a discrete color palette as an independent, controllable conditioning signal—decoupled from textual prompts—into the diffusion generation process. We construct the first large-scale palette–text–image triplet dataset and systematically investigate diverse palette representations, including RGB vectors, k-means cluster centers, and learnable embeddings, enabling joint conditional modeling of text and color. Experiments demonstrate significant improvements in color accuracy, global chromatic harmony, and visual fidelity under target palettes, especially for background and secondary object regions. Our approach establishes a new paradigm for fine-grained, interpretable, and controllable image coloring.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

With the advent of diffusion models, Text-to-Image (T2I) generation has seen substantial advancements. Current T2I models allow users to specify object colors using linguistic color names, and some methods aim to personalize color-object association through prompt learning. However, existing models struggle to provide comprehensive control over the color schemes of an entire image, especially for background elements and less prominent objects not explicitly mentioned in prompts. This paper proposes a novel approach to enhance color scheme control by integrating color palettes as a separate guidance mechanism alongside prompt instructions. We investigate the effectiveness of palette guidance by exploring various palette representation methods within a diffusion-based image colorization framework. To facilitate this exploration, we construct specialized palette-text-image datasets and conduct extensive quantitative and qualitative analyses. Our results demonstrate that incorporating palette guidance significantly improves the model's ability to generate images with desired color schemes, enabling a more controlled and refined colorization process.

Problem

• Enhancing color scheme control in diffusion models
• Addressing limited color control for background elements
• Integrating color palettes as guidance mechanism

Innovation

• Integrates color palettes as separate guidance mechanism
• Explores various palette representation methods
• Constructs specialized palette-text-image datasets